Jul 02 20:14:50.890510 ip-10-0-132-15 systemd[1]: Starting Kubernetes Kubelet... Jul 02 20:14:51.292082 ip-10-0-132-15 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 20:14:51.292082 ip-10-0-132-15 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 20:14:51.292082 ip-10-0-132-15 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 20:14:51.292082 ip-10-0-132-15 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 20:14:51.292082 ip-10-0-132-15 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 20:14:51.293667 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.293580 2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 20:14:51.297180 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297164 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.297180 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297180 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297183 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297188 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297191 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297194 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297197 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297200 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297203 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297219 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297222 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297224 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297227 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297230 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297232 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297235 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297238 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297240 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297243 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297245 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297254 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.297296 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297257 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297259 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297262 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297265 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297267 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297270 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297272 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297274 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297277 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297279 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297281 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297284 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297286 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297289 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297292 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297295 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297297 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297300 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297303 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.297768 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297305 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297310 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297314 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297317 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297320 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297322 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297325 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297328 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297330 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297333 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297335 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297338 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297340 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297343 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297345 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297348 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297350 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297353 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297356 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297358 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.298260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297361 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297364 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297366 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297369 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297372 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297374 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297377 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297380 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297383 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297387 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297390 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297393 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297396 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297398 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297401 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297403 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297406 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297408 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297410 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.298747 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297413 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297415 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297419 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297421 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297424 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297426 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297429 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297431 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297434 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297436 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297439 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297441 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297444 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297446 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297450 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297452 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297455 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297459 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297462 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.299218 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.297464 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299000 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299007 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299011 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299013 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299017 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299019 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299022 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299025 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299028 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299031 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299034 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299036 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299039 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299041 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299044 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299047 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299050 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299052 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299055 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.299779 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299057 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299061 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299063 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299066 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299068 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299072 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299076 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299078 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299081 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299083 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299085 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299089 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299091 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299094 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299096 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299100 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299103 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299106 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299109 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299111 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.300630 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299114 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299117 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299119 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299121 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299124 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299126 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299129 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299131 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299134 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299137 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299139 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299142 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299145 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299148 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299151 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299153 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299156 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299159 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299161 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299163 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.301146 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299166 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299168 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299170 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299173 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299175 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299178 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299181 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299183 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299186 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299188 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299191 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299193 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299196 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299199 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299201 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299203 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299221 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299224 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299228 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.301655 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299232 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299235 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299238 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299241 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299245 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299247 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299250 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299252 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299255 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299258 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299261 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299264 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299267 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299269 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299271 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299274 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299276 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299278 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299282 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299284 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.302124 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.299287 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301275 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301284 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301288 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301294 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301297 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301300 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301302 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301305 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301307 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301310 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301313 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301316 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301318 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301322 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301325 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301330 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301333 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301336 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.302664 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301338 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301342 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301344 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301347 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301350 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301352 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301355 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301358 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301360 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301363 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301368 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301370 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301373 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301376 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301378 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301381 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301384 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301387 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301390 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301393 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.303120 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301395 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301398 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301403 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301406 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301408 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301411 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301413 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301418 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301421 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301426 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301429 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301431 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301434 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301437 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301443 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301446 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301448 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301451 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301453 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.303635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301456 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301458 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301461 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301463 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301467 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301470 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301472 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301478 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301480 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301482 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301485 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301488 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301491 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301495 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301499 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301502 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301505 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301508 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301513 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.304089 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301519 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301522 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301524 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301527 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301529 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301532 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301534 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301537 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301540 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301543 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301545 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301548 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301553 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301555 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301557 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301560 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301563 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301565 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301568 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301571 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.304560 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301573 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301576 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.301578 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301801 2565 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301819 2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301830 2565 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301834 2565 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301946 2565 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301952 2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301956 2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301961 2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301964 2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301967 2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301970 2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301974 2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301977 2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301980 2565 flags.go:64] FLAG: --cgroup-root="" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301983 2565 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301985 2565 flags.go:64] FLAG: --client-ca-file="" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301988 2565 flags.go:64] FLAG: --cloud-provider="external" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301991 2565 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301996 2565 flags.go:64] FLAG: --cluster-domain="" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.301998 2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302001 2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 20:14:51.305064 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302005 2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302008 2565 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302015 2565 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302018 2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302021 2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302025 2565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302028 2565 flags.go:64] FLAG: --contention-profiling="false" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302031 2565 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302034 2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302037 2565 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302040 2565 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302044 2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302047 2565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302050 2565 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302053 2565 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302056 2565 flags.go:64] FLAG: --enable-server="true" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302059 2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302065 2565 flags.go:64] FLAG: --event-burst="100" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302068 2565 flags.go:64] FLAG: --event-qps="50" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302071 2565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302074 2565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302077 2565 flags.go:64] FLAG: --eviction-hard="" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302080 2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302083 2565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302086 2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 20:14:51.305688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302089 2565 flags.go:64] FLAG: --eviction-soft="" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302092 2565 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302095 2565 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302097 2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302100 2565 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302103 2565 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302106 2565 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302108 2565 flags.go:64] FLAG: --feature-gates="" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302112 2565 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302115 2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302118 2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302121 2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302125 2565 flags.go:64] FLAG: --healthz-port="10248" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302128 2565 flags.go:64] FLAG: --help="false" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302131 2565 flags.go:64] FLAG: --hostname-override="ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302134 2565 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302136 2565 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302139 2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302143 2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302148 2565 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302151 2565 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302154 2565 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302156 2565 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 20:14:51.306343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302159 2565 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302162 2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302165 2565 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302168 2565 flags.go:64] FLAG: --kube-reserved="" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302171 2565 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302173 2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302177 2565 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302180 2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302182 2565 flags.go:64] FLAG: --lock-file="" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302185 2565 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302188 2565 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302190 2565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302196 2565 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302198 2565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302201 2565 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302204 2565 flags.go:64] FLAG: --logging-format="text" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302223 2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302228 2565 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302233 2565 flags.go:64] FLAG: --manifest-url="" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302237 2565 flags.go:64] FLAG: --manifest-url-header="" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302242 2565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302245 2565 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302250 2565 flags.go:64] FLAG: --max-pods="110" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302253 2565 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302256 2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 20:14:51.306925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302259 2565 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302262 2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302265 2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302268 2565 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302271 2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302279 2565 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302282 2565 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302285 2565 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302288 2565 flags.go:64] FLAG: --pod-cidr="" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302291 2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302296 2565 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302299 2565 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302302 2565 flags.go:64] FLAG: --pods-per-core="0" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302305 2565 flags.go:64] FLAG: --port="10250" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302308 2565 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302311 2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0075bda412b5c7936" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302314 2565 flags.go:64] FLAG: --qos-reserved="" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302317 2565 flags.go:64] FLAG: --read-only-port="10255" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302319 2565 flags.go:64] FLAG: --register-node="true" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302322 2565 flags.go:64] FLAG: --register-with-taints="" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302325 2565 flags.go:64] FLAG: --registry-burst="10" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302328 2565 flags.go:64] FLAG: --registry-qps="5" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302332 2565 flags.go:64] FLAG: --reserved-cpus="" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302335 2565 flags.go:64] FLAG: --reserved-memory="" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302338 2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 20:14:51.307542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302342 2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302345 2565 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302348 2565 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302350 2565 flags.go:64] FLAG: --runonce="false" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302353 2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302356 2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302359 2565 flags.go:64] FLAG: --seccomp-default="false" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302361 2565 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302364 2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302367 2565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302370 2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302376 2565 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302379 2565 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302382 2565 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302385 2565 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302388 2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302391 2565 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302393 2565 flags.go:64] FLAG: --system-cgroups="" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302396 2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302402 2565 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302405 2565 flags.go:64] FLAG: --tls-cert-file="" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302408 2565 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302412 2565 flags.go:64] FLAG: --tls-min-version="" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302415 2565 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302418 2565 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 20:14:51.308145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302421 2565 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302424 2565 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302427 2565 flags.go:64] FLAG: --v="2" Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302431 2565 flags.go:64] FLAG: --version="false" Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302435 2565 flags.go:64] FLAG: --vmodule="" Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302439 2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302442 2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302586 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302591 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302594 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302597 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302599 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302602 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302605 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302607 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302610 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302612 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302615 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302618 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302621 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302624 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.309106 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302627 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302629 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302632 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302635 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302637 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302640 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302642 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302644 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302647 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302649 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302652 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302654 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302656 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302659 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302661 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302664 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302666 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302669 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302672 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302675 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.310043 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302677 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302679 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302682 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302684 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302686 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302689 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302691 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302694 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302698 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302702 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302706 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302709 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302711 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302714 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302716 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302719 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302721 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302724 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302727 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.310906 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302729 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302731 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302734 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302736 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302738 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302741 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302743 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302746 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302748 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302751 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302753 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302756 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302758 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302761 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302763 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302766 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302768 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302770 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302773 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.311857 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302776 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302779 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302783 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302787 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302789 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302792 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302795 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302797 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302800 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302803 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302805 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302807 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302810 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302813 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302815 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302818 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302820 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302822 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302825 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302827 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.312457 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302830 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302832 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302834 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302837 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302839 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302842 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.302846 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.302851 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.310606 2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.310626 2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.310650 2565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.310658 2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310742 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310751 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.313173 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310756 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310761 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310765 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310770 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310774 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310778 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310782 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310786 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310790 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310795 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310799 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310803 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310808 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310815 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310821 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310826 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310830 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310836 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310841 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310845 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.313801 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310849 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310853 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310857 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310861 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310866 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310870 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310874 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310883 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310889 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310893 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310898 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310902 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310906 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310910 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310913 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310917 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310921 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310925 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310929 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.314458 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310933 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310937 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310941 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310945 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310950 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310954 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310958 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310962 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310967 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310971 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310975 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310979 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310983 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310987 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310991 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.310996 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311000 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311004 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311008 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.315044 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311012 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311018 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311023 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311028 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311032 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311036 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311040 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311044 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311048 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311053 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311057 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311061 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311065 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311070 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311074 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311078 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311083 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311087 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311091 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.315558 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311095 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311099 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311103 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311107 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311111 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311116 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311121 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311127 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311132 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311136 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311140 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311144 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311148 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311152 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311157 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311161 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311167 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311171 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311175 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.316010 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311178 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.311186 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311376 2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311386 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311391 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311395 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311399 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311404 2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311408 2565 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311412 2565 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311416 2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311421 2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311426 2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311432 2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311436 2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 20:14:51.316537 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311441 2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311445 2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311449 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311453 2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311459 2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311464 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311469 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311473 2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311478 2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311482 2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311486 2565 feature_gate.go:328] unrecognized feature gate: Example Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311490 2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311495 2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311500 2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311503 2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311509 2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311513 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311517 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311522 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311526 2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 20:14:51.316923 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311530 2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311533 2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311537 2565 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311541 2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311545 2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311549 2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311553 2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311557 2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311561 2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311565 2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311569 2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311573 2565 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311577 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311581 2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311584 2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311588 2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311592 2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311596 2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311600 2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311604 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 20:14:51.317474 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311607 2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311614 2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311620 2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311625 2565 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311630 2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311635 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311640 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311644 2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311649 2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311654 2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311658 2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311662 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311666 2565 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311670 2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311674 2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311678 2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311682 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311686 2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311690 2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 20:14:51.317984 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311694 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311698 2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311702 2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311706 2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311710 2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311714 2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311718 2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311722 2565 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311727 2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311730 2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311734 2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311739 2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311743 2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311747 2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311751 2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311756 2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311760 2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311764 2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311768 2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311772 2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 20:14:51.318764 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311775 2565 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311780 2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311785 2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311789 2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311793 2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311797 2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:51.311801 2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.311809 2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.311974 2565 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.314634 2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.316238 2565 server.go:1019] "Starting client certificate rotation" Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.316343 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.316373 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 20:14:51.319576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.318785 2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 20:14:51.320497 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.320477 2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 20:14:51.333809 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.333789 2565 log.go:25] "Validated CRI v1 runtime API" Jul 02 20:14:51.336811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.336795 2565 log.go:25] "Validated CRI v1 image API" Jul 02 20:14:51.337567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.337554 2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 20:14:51.340228 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.340187 2565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81c753da-781a-4ea6-bcd8-7dd568782a02:/dev/nvme0n1p3 e89d974b-f3bd-4d60-8f9e-9a11caad1e7e:/dev/nvme0n1p4] Jul 02 20:14:51.340291 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.340230 2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 20:14:51.345314 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.345299 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 20:14:51.345579 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.345470 2565 manager.go:216] Machine: {Timestamp:2026-07-02 20:14:51.344224017 +0000 UTC m=+0.340527241 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106881 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2324c55a9542c635e329bd17f5b9cc SystemUUID:ec2324c5-5a95-42c6-35e3-29bd17f5b9cc BootID:4128bd97-ec89-43be-93e9-787676b2577e Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:28:d6:e5:31:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:28:d6:e5:31:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:41:a3:95:27:44 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 20:14:51.345579 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.345578 2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 20:14:51.345679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.345667 2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 20:14:51.349285 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.349255 2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 20:14:51.349417 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.349289 2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 20:14:51.350486 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.350475 2565 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 20:14:51.350518 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.350488 2565 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 20:14:51.350518 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.350496 2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 20:14:51.351152 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.351142 2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 20:14:51.352687 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.352677 2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 20:14:51.355888 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.355877 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 02 20:14:51.356182 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.356172 2565 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 20:14:51.356228 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.356219 2565 kubelet.go:486] "Attempting to sync node with API server" Jul 02 20:14:51.356261 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.356229 2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 20:14:51.356261 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.356244 2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 20:14:51.356261 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.356252 2565 kubelet.go:398] "Adding apiserver pod source" Jul 02 20:14:51.356261 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.356261 2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 20:14:51.356570 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.356560 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 20:14:51.356615 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.356576 2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 20:14:51.360204 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.360185 2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 20:14:51.362058 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.362045 2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 20:14:51.362120 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.362074 2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 20:14:51.363256 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363242 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 20:14:51.363256 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363260 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 20:14:51.363370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363266 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 20:14:51.363370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363272 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 20:14:51.363370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363282 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 20:14:51.363370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363290 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 20:14:51.363370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363301 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 20:14:51.363370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363308 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 20:14:51.363370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363315 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 20:14:51.363370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363321 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 20:14:51.363370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363329 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 20:14:51.363370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363334 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 20:14:51.363997 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363987 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 20:14:51.363997 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.363997 2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 20:14:51.367405 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.367390 2565 server.go:1268] "Started kubelet" Jul 02 20:14:51.367517 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.367490 2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 20:14:51.367616 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.367560 2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 20:14:51.367720 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.367632 2565 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 20:14:51.367829 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.367809 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4p5wg" Jul 02 20:14:51.368460 ip-10-0-132-15 systemd[1]: Started Kubernetes Kubelet. Jul 02 20:14:51.369285 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.369269 2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-15.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 20:14:51.369285 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.369274 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 20:14:51.369285 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.369266 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 20:14:51.369730 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.369717 2565 server.go:310] "Adding debug handlers to kubelet server" Jul 02 20:14:51.372304 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.372286 2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 20:14:51.373174 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.373155 2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 20:14:51.374242 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.373391 2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-15.ec2.internal.18be923c94331a05 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-15.ec2.internal,UID:ip-10-0-132-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-15.ec2.internal,},FirstTimestamp:2026-07-02 20:14:51.367365125 +0000 UTC m=+0.363668348,LastTimestamp:2026-07-02 20:14:51.367365125 +0000 UTC m=+0.363668348,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-15.ec2.internal,}" Jul 02 20:14:51.375697 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.375682 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4p5wg" Jul 02 20:14:51.377028 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.377015 2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 20:14:51.377079 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.377027 2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 20:14:51.377122 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.377074 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:51.377900 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.377861 2565 factory.go:55] Registering systemd factory Jul 02 20:14:51.377900 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.377884 2565 factory.go:223] Registration of the systemd container factory successfully Jul 02 20:14:51.378334 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.378189 2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 20:14:51.378334 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.378178 2565 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 20:14:51.378439 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.378348 2565 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 20:14:51.378533 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.378518 2565 reconstruct.go:97] "Volume reconstruction finished" Jul 02 20:14:51.378533 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.378533 2565 reconciler.go:29] "Reconciler: start to sync state" Jul 02 20:14:51.378656 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.378573 2565 factory.go:153] Registering CRI-O factory Jul 02 20:14:51.378656 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.378589 2565 factory.go:223] Registration of the crio container factory successfully Jul 02 20:14:51.378656 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.378640 2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 20:14:51.378800 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.378664 2565 factory.go:103] Registering Raw factory Jul 02 20:14:51.378800 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.378681 2565 manager.go:1196] Started watching for new ooms in manager Jul 02 20:14:51.381745 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.381721 2565 manager.go:349] Starting recovery of all containers Jul 02 20:14:51.382515 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.382488 2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 02 20:14:51.382610 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.382591 2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 02 20:14:51.386996 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.386963 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 20:14:51.392377 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.392030 2565 manager.go:354] Recovery completed Jul 02 20:14:51.396627 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.396614 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.398817 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.398798 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.398892 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.398832 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.398892 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.398846 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.399408 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.399395 2565 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 20:14:51.399408 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.399407 2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 20:14:51.399498 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.399410 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.399498 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.399423 2565 state_mem.go:36] "Initialized new in-memory state store" Jul 02 20:14:51.400185 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.400169 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.400260 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.400198 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.400260 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.400221 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.400800 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.400788 2565 policy_none.go:49] "None policy: Start" Jul 02 20:14:51.400840 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.400804 2565 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 20:14:51.400840 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.400813 2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 20:14:51.401833 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.401823 2565 policy_none.go:47] "Start" Jul 02 20:14:51.444860 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.444810 2565 manager.go:337] "Starting Device Plugin manager" Jul 02 20:14:51.444860 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.444857 2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 20:14:51.445002 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.444871 2565 server.go:87] "Starting device plugin registration server" Jul 02 20:14:51.445137 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.445127 2565 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 20:14:51.445173 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.445139 2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 20:14:51.445339 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.445322 2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 20:14:51.445418 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.445404 2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 20:14:51.445418 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.445413 2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 20:14:51.446355 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.446338 2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 20:14:51.446444 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.446381 2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:51.545801 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.545725 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.547568 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.547550 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.547639 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.547581 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.547639 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.547592 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.547639 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.547624 2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.555706 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.555683 2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.555765 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.555709 2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-15.ec2.internal\": node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:51.564745 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.564723 2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 20:14:51.564838 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.564760 2565 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 20:14:51.564838 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.564778 2565 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 20:14:51.564838 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.564804 2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 20:14:51.565371 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.565288 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:51.567253 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.567234 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:14:51.665083 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.665033 2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-15.ec2.internal"] Jul 02 20:14:51.665246 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.665124 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.665346 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.665330 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:51.670642 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.670625 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.670755 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.670670 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.670755 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.670685 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.672198 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.672182 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.672314 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.672293 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.672362 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.672318 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.673119 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.673104 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.673191 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.673126 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.673191 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.673137 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.673191 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.673108 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.673191 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.673168 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.673191 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.673178 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.674493 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.674477 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.674552 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.674506 2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 20:14:51.675364 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.675349 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientMemory" Jul 02 20:14:51.675427 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.675375 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 20:14:51.675427 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.675387 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeHasSufficientPID" Jul 02 20:14:51.680794 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.680774 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99ae426d085798cd8010de0c176c4766-config\") pod \"kube-apiserver-proxy-ip-10-0-132-15.ec2.internal\" (UID: \"99ae426d085798cd8010de0c176c4766\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.680891 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.680799 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70902e0777cf94797571da24b10ed602-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal\" (UID: \"70902e0777cf94797571da24b10ed602\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.680891 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.680815 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70902e0777cf94797571da24b10ed602-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal\" (UID: \"70902e0777cf94797571da24b10ed602\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.691808 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.691790 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-15.ec2.internal\" not found" node="ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.696185 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.696168 2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-15.ec2.internal\" not found" node="ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.765459 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.765431 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:51.781163 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.781141 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70902e0777cf94797571da24b10ed602-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal\" (UID: \"70902e0777cf94797571da24b10ed602\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.781292 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.781168 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70902e0777cf94797571da24b10ed602-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal\" (UID: \"70902e0777cf94797571da24b10ed602\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.781292 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.781185 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99ae426d085798cd8010de0c176c4766-config\") pod \"kube-apiserver-proxy-ip-10-0-132-15.ec2.internal\" (UID: \"99ae426d085798cd8010de0c176c4766\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.781292 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.781230 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/99ae426d085798cd8010de0c176c4766-config\") pod \"kube-apiserver-proxy-ip-10-0-132-15.ec2.internal\" (UID: \"99ae426d085798cd8010de0c176c4766\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.781292 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.781254 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/70902e0777cf94797571da24b10ed602-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal\" (UID: \"70902e0777cf94797571da24b10ed602\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.781292 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.781261 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70902e0777cf94797571da24b10ed602-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal\" (UID: \"70902e0777cf94797571da24b10ed602\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.866239 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.866149 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:51.966263 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:51.966229 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:51.994775 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.994745 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" Jul 02 20:14:51.999509 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:51.999495 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-15.ec2.internal" Jul 02 20:14:52.066862 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:52.066835 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:52.173071 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:52.172968 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:52.275477 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:52.275442 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:52.317167 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.317144 2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 20:14:52.317758 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.317295 2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 20:14:52.375684 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:52.375657 2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-15.ec2.internal\" not found" Jul 02 20:14:52.382322 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.382283 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 20:09:52 +0000 UTC" deadline="2028-01-02 11:57:17.605929273 +0000 UTC" Jul 02 20:14:52.382322 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.382319 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13167h42m25.223613175s" Jul 02 20:14:52.420638 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.420611 2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:14:52.477374 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.477318 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" Jul 02 20:14:52.489551 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.489528 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 20:14:52.493121 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.493099 2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-15.ec2.internal" Jul 02 20:14:52.500770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.500751 2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 20:14:52.556013 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:52.555977 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70902e0777cf94797571da24b10ed602.slice/crio-5e63d0e7842acb92146f7c8933c12ac02e47eb03e99b88cd0919c3ae5ca24587 WatchSource:0}: Error finding container 5e63d0e7842acb92146f7c8933c12ac02e47eb03e99b88cd0919c3ae5ca24587: Status 404 returned error can't find the container with id 5e63d0e7842acb92146f7c8933c12ac02e47eb03e99b88cd0919c3ae5ca24587 Jul 02 20:14:52.556278 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:52.556254 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ae426d085798cd8010de0c176c4766.slice/crio-4bc55d06d3a35c06b11bb88d1019beb64672b0a12a255e85c357ec442f027e10 WatchSource:0}: Error finding container 4bc55d06d3a35c06b11bb88d1019beb64672b0a12a255e85c357ec442f027e10: Status 404 returned error can't find the container with id 4bc55d06d3a35c06b11bb88d1019beb64672b0a12a255e85c357ec442f027e10 Jul 02 20:14:52.561340 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.561324 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:14:52.568674 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.568631 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-15.ec2.internal" event={"ID":"99ae426d085798cd8010de0c176c4766","Type":"ContainerStarted","Data":"4bc55d06d3a35c06b11bb88d1019beb64672b0a12a255e85c357ec442f027e10"} Jul 02 20:14:52.569804 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.569782 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" event={"ID":"70902e0777cf94797571da24b10ed602","Type":"ContainerStarted","Data":"5e63d0e7842acb92146f7c8933c12ac02e47eb03e99b88cd0919c3ae5ca24587"} Jul 02 20:14:52.746929 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.746844 2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:14:52.950124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:52.950088 2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:14:53.358921 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.358877 2565 apiserver.go:52] "Watching apiserver" Jul 02 20:14:53.370404 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.370066 2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 20:14:53.371753 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.371718 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-5c7nw","openshift-multus/multus-nswz8","openshift-multus/network-metrics-daemon-hr888","openshift-network-operator/iptables-alerter-shnxp","kube-system/konnectivity-agent-hd7jg","kube-system/kube-apiserver-proxy-ip-10-0-132-15.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t","openshift-cluster-node-tuning-operator/tuned-g8mkj","openshift-network-diagnostics/network-check-target-764hb","openshift-ovn-kubernetes/ovnkube-node-h6ssd","kube-system/global-pull-secret-syncer-nkzfc","openshift-dns/node-resolver-8thwh","openshift-image-registry/node-ca-jz4g9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal"] Jul 02 20:14:53.373321 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.373300 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.374605 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.374576 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jz4g9" Jul 02 20:14:53.376131 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.376106 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:53.376228 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.376189 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:14:53.377199 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.377176 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 20:14:53.377322 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.377193 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 20:14:53.377322 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.377188 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 20:14:53.377443 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.377325 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5srvv\"" Jul 02 20:14:53.378210 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.377681 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 20:14:53.378210 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.377714 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-shnxp" Jul 02 20:14:53.378210 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.377744 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 20:14:53.378210 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.377691 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ltlnm\"" Jul 02 20:14:53.378210 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.377966 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 20:14:53.378210 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.378090 2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 20:14:53.379017 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.378999 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:14:53.380705 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.380676 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 20:14:53.380835 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.380813 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-24z9b\"" Jul 02 20:14:53.381426 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.381388 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.383894 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.383654 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 20:14:53.384354 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.384180 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pwzwz\"" Jul 02 20:14:53.384354 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.384312 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 20:14:53.387460 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.385177 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 20:14:53.387460 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.385493 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 20:14:53.387460 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.387197 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 20:14:53.387460 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.387207 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tkbjk\"" Jul 02 20:14:53.387885 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.387866 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388461 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdgpm\" (UniqueName: \"kubernetes.io/projected/ded82f6b-f240-4500-9e22-bcc20618b526-kube-api-access-gdgpm\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9ef4988-0ad9-491c-9935-929c5a0eb8ba-host\") pod \"node-ca-jz4g9\" (UID: \"a9ef4988-0ad9-491c-9935-929c5a0eb8ba\") " pod="openshift-image-registry/node-ca-jz4g9" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388515 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388530 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388559 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3aae5d08-0b74-4611-8932-d1afc7b80412-agent-certs\") pod \"konnectivity-agent-hd7jg\" (UID: \"3aae5d08-0b74-4611-8932-d1afc7b80412\") " pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388585 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3aae5d08-0b74-4611-8932-d1afc7b80412-konnectivity-ca\") pod \"konnectivity-agent-hd7jg\" (UID: \"3aae5d08-0b74-4611-8932-d1afc7b80412\") " pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.388580 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388614 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnsvr\" (UniqueName: \"kubernetes.io/projected/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-kube-api-access-lnsvr\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8180e463-918f-413b-ae8f-5ceabe81e0fe-iptables-alerter-script\") pod \"iptables-alerter-shnxp\" (UID: \"8180e463-918f-413b-ae8f-5ceabe81e0fe\") " pod="openshift-network-operator/iptables-alerter-shnxp" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388663 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8180e463-918f-413b-ae8f-5ceabe81e0fe-host-slash\") pod \"iptables-alerter-shnxp\" (UID: \"8180e463-918f-413b-ae8f-5ceabe81e0fe\") " pod="openshift-network-operator/iptables-alerter-shnxp" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388688 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-etc-selinux\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388712 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzd2k\" (UniqueName: \"kubernetes.io/projected/a9ef4988-0ad9-491c-9935-929c5a0eb8ba-kube-api-access-pzd2k\") pod \"node-ca-jz4g9\" (UID: \"a9ef4988-0ad9-491c-9935-929c5a0eb8ba\") " pod="openshift-image-registry/node-ca-jz4g9" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388736 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-registration-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388758 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-device-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.388781 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388781 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9ef4988-0ad9-491c-9935-929c5a0eb8ba-serviceca\") pod \"node-ca-jz4g9\" (UID: \"a9ef4988-0ad9-491c-9935-929c5a0eb8ba\") " pod="openshift-image-registry/node-ca-jz4g9" Jul 02 20:14:53.389580 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388805 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfp42\" (UniqueName: \"kubernetes.io/projected/8180e463-918f-413b-ae8f-5ceabe81e0fe-kube-api-access-zfp42\") pod \"iptables-alerter-shnxp\" (UID: \"8180e463-918f-413b-ae8f-5ceabe81e0fe\") " pod="openshift-network-operator/iptables-alerter-shnxp" Jul 02 20:14:53.389580 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.389580 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388851 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-socket-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.389580 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.388895 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-sys-fs\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.390591 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.390043 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.391433 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.391414 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.392664 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.392648 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rchhd\"" Jul 02 20:14:53.392991 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.392968 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.393082 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.393067 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 20:14:53.393353 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.393285 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 20:14:53.393738 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.393720 2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 20:14:53.394513 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.394103 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 20:14:53.394513 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.394222 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-t72zp\"" Jul 02 20:14:53.394513 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.394111 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 20:14:53.394513 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.394339 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 20:14:53.394513 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.394111 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 20:14:53.394901 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.394738 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:53.394901 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.394795 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:14:53.396074 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.396053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8thwh" Jul 02 20:14:53.397145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.396916 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2ts52\"" Jul 02 20:14:53.397223 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.397161 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 20:14:53.397413 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.397397 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 20:14:53.397967 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.397946 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 20:14:53.398259 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.398230 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 20:14:53.398533 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.398513 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 20:14:53.398796 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.398778 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 20:14:53.399696 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.399675 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 20:14:53.399970 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.399953 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 20:14:53.400197 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.400182 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-96pdj\"" Jul 02 20:14:53.416603 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.416576 2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tp9sn" Jul 02 20:14:53.424753 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.424364 2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tp9sn" Jul 02 20:14:53.480054 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.480028 2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 20:14:53.489103 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489069 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-run\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.489247 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9ef4988-0ad9-491c-9935-929c5a0eb8ba-host\") pod \"node-ca-jz4g9\" (UID: \"a9ef4988-0ad9-491c-9935-929c5a0eb8ba\") " pod="openshift-image-registry/node-ca-jz4g9" Jul 02 20:14:53.489247 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-tuned\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.489247 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-etc-kubernetes\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.489247 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-sysctl-d\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.489247 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489221 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-var-lib-kubelet\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489264 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa235a05-22fe-4e3a-adc5-705913aad0c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489286 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-var-lib-openvswitch\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489318 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-run-openvswitch\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489344 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-kubelet-config\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489372 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-etc-selinux\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489398 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa235a05-22fe-4e3a-adc5-705913aad0c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489423 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d462c3b4-ffb3-4449-aff4-cd348fbc835c-tmp-dir\") pod \"node-resolver-8thwh\" (UID: \"d462c3b4-ffb3-4449-aff4-cd348fbc835c\") " pod="openshift-dns/node-resolver-8thwh" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489449 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-device-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489475 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-cnibin\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-run-netns\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489531 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f0c99d5-da5d-434c-99fe-762eff122a1e-ovn-node-metrics-cert\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.489567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489556 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66trh\" (UniqueName: \"kubernetes.io/projected/d462c3b4-ffb3-4449-aff4-cd348fbc835c-kube-api-access-66trh\") pod \"node-resolver-8thwh\" (UID: \"d462c3b4-ffb3-4449-aff4-cd348fbc835c\") " pod="openshift-dns/node-resolver-8thwh" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489581 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-var-lib-cni-multus\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn22x\" (UniqueName: \"kubernetes.io/projected/0b7cafcc-8972-4475-8f86-4e22f41d1604-kube-api-access-bn22x\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489635 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-cnibin\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489659 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-socket-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489683 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-sys-fs\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdgpm\" (UniqueName: \"kubernetes.io/projected/ded82f6b-f240-4500-9e22-bcc20618b526-kube-api-access-gdgpm\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489720 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-kubernetes\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489741 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-systemd\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489769 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnsvr\" (UniqueName: \"kubernetes.io/projected/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-kube-api-access-lnsvr\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489787 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8180e463-918f-413b-ae8f-5ceabe81e0fe-iptables-alerter-script\") pod \"iptables-alerter-shnxp\" (UID: \"8180e463-918f-413b-ae8f-5ceabe81e0fe\") " pod="openshift-network-operator/iptables-alerter-shnxp" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489840 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvjbm\" (UniqueName: \"kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm\") pod \"network-check-target-764hb\" (UID: \"57d801c5-6014-4d74-a071-b8ead03d72d1\") " pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489867 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b7cafcc-8972-4475-8f86-4e22f41d1604-cni-binary-copy\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489894 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa235a05-22fe-4e3a-adc5-705913aad0c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489919 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-run-ovn\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489954 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-node-log\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.490124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.489981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6797\" (UniqueName: \"kubernetes.io/projected/fa235a05-22fe-4e3a-adc5-705913aad0c4-kube-api-access-x6797\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490017 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490042 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f0c99d5-da5d-434c-99fe-762eff122a1e-ovnkube-script-lib\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490067 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dbc6243-cea2-42c1-88da-d3c44128342e-tmp\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490127 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-system-cni-dir\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490149 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-systemd-units\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490175 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-run-netns\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490199 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-log-socket\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490237 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-cni-netd\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490311 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3aae5d08-0b74-4611-8932-d1afc7b80412-konnectivity-ca\") pod \"konnectivity-agent-hd7jg\" (UID: \"3aae5d08-0b74-4611-8932-d1afc7b80412\") " pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490405 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-kubelet\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490521 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-etc-selinux\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490626 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-sys-fs\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.490709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490684 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a9ef4988-0ad9-491c-9935-929c5a0eb8ba-host\") pod \"node-ca-jz4g9\" (UID: \"a9ef4988-0ad9-491c-9935-929c5a0eb8ba\") " pod="openshift-image-registry/node-ca-jz4g9" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.490819 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.490940 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs podName:8caf8e2d-384f-4ecb-a60c-3ce8137a419a nodeName:}" failed. No retries permitted until 2026-07-02 20:14:53.990903557 +0000 UTC m=+2.987206791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs") pod "network-metrics-daemon-hr888" (UID: "8caf8e2d-384f-4ecb-a60c-3ce8137a419a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.490941 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-socket-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491016 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-device-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491057 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8180e463-918f-413b-ae8f-5ceabe81e0fe-host-slash\") pod \"iptables-alerter-shnxp\" (UID: \"8180e463-918f-413b-ae8f-5ceabe81e0fe\") " pod="openshift-network-operator/iptables-alerter-shnxp" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491128 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-var-lib-cni-bin\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-daemon-config\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491190 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491209 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-slash\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491236 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzd2k\" (UniqueName: \"kubernetes.io/projected/a9ef4988-0ad9-491c-9935-929c5a0eb8ba-kube-api-access-pzd2k\") pod \"node-ca-jz4g9\" (UID: \"a9ef4988-0ad9-491c-9935-929c5a0eb8ba\") " pod="openshift-image-registry/node-ca-jz4g9" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-lib-modules\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491346 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-run-systemd\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491385 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f0c99d5-da5d-434c-99fe-762eff122a1e-env-overrides\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491416 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-dbus\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:53.491462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491452 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-registration-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491481 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp42\" (UniqueName: \"kubernetes.io/projected/8180e463-918f-413b-ae8f-5ceabe81e0fe-kube-api-access-zfp42\") pod \"iptables-alerter-shnxp\" (UID: \"8180e463-918f-413b-ae8f-5ceabe81e0fe\") " pod="openshift-network-operator/iptables-alerter-shnxp" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491513 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f0c99d5-da5d-434c-99fe-762eff122a1e-ovnkube-config\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491539 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-sys\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491570 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3aae5d08-0b74-4611-8932-d1afc7b80412-agent-certs\") pod \"konnectivity-agent-hd7jg\" (UID: \"3aae5d08-0b74-4611-8932-d1afc7b80412\") " pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491631 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-cni-dir\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-os-release\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491787 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-run-k8s-cni-cncf-io\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491790 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8180e463-918f-413b-ae8f-5ceabe81e0fe-iptables-alerter-script\") pod \"iptables-alerter-shnxp\" (UID: \"8180e463-918f-413b-ae8f-5ceabe81e0fe\") " pod="openshift-network-operator/iptables-alerter-shnxp" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491823 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-etc-openvswitch\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491856 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fgg8\" (UniqueName: \"kubernetes.io/projected/2f0c99d5-da5d-434c-99fe-762eff122a1e-kube-api-access-8fgg8\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491883 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-sysconfig\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491909 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3aae5d08-0b74-4611-8932-d1afc7b80412-konnectivity-ca\") pod \"konnectivity-agent-hd7jg\" (UID: \"3aae5d08-0b74-4611-8932-d1afc7b80412\") " pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491914 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-registration-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.491981 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-host\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492018 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ded82f6b-f240-4500-9e22-bcc20618b526-kubelet-dir\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.492237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492019 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-run-multus-certs\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492148 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-system-cni-dir\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492179 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8180e463-918f-413b-ae8f-5ceabe81e0fe-host-slash\") pod \"iptables-alerter-shnxp\" (UID: \"8180e463-918f-413b-ae8f-5ceabe81e0fe\") " pod="openshift-network-operator/iptables-alerter-shnxp" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492182 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-cni-bin\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492220 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d462c3b4-ffb3-4449-aff4-cd348fbc835c-hosts-file\") pod \"node-resolver-8thwh\" (UID: \"d462c3b4-ffb3-4449-aff4-cd348fbc835c\") " pod="openshift-dns/node-resolver-8thwh" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-sysctl-conf\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492271 2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492332 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-socket-dir-parent\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492379 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-var-lib-kubelet\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492431 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-hostroot\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492472 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-conf-dir\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492544 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9ef4988-0ad9-491c-9935-929c5a0eb8ba-serviceca\") pod \"node-ca-jz4g9\" (UID: \"a9ef4988-0ad9-491c-9935-929c5a0eb8ba\") " pod="openshift-image-registry/node-ca-jz4g9" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492600 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-modprobe-d\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492695 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh95n\" (UniqueName: \"kubernetes.io/projected/6dbc6243-cea2-42c1-88da-d3c44128342e-kube-api-access-gh95n\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.492913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492752 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-os-release\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.493410 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.492991 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a9ef4988-0ad9-491c-9935-929c5a0eb8ba-serviceca\") pod \"node-ca-jz4g9\" (UID: \"a9ef4988-0ad9-491c-9935-929c5a0eb8ba\") " pod="openshift-image-registry/node-ca-jz4g9" Jul 02 20:14:53.499855 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.496424 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3aae5d08-0b74-4611-8932-d1afc7b80412-agent-certs\") pod \"konnectivity-agent-hd7jg\" (UID: \"3aae5d08-0b74-4611-8932-d1afc7b80412\") " pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:14:53.506272 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.502267 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnsvr\" (UniqueName: \"kubernetes.io/projected/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-kube-api-access-lnsvr\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:53.506272 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.503126 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfp42\" (UniqueName: \"kubernetes.io/projected/8180e463-918f-413b-ae8f-5ceabe81e0fe-kube-api-access-zfp42\") pod \"iptables-alerter-shnxp\" (UID: \"8180e463-918f-413b-ae8f-5ceabe81e0fe\") " pod="openshift-network-operator/iptables-alerter-shnxp" Jul 02 20:14:53.506272 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.503606 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzd2k\" (UniqueName: \"kubernetes.io/projected/a9ef4988-0ad9-491c-9935-929c5a0eb8ba-kube-api-access-pzd2k\") pod \"node-ca-jz4g9\" (UID: \"a9ef4988-0ad9-491c-9935-929c5a0eb8ba\") " pod="openshift-image-registry/node-ca-jz4g9" Jul 02 20:14:53.506272 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.503620 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdgpm\" (UniqueName: \"kubernetes.io/projected/ded82f6b-f240-4500-9e22-bcc20618b526-kube-api-access-gdgpm\") pod \"aws-ebs-csi-driver-node-9d78t\" (UID: \"ded82f6b-f240-4500-9e22-bcc20618b526\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.593096 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593024 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f0c99d5-da5d-434c-99fe-762eff122a1e-ovn-node-metrics-cert\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.593302 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593113 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66trh\" (UniqueName: \"kubernetes.io/projected/d462c3b4-ffb3-4449-aff4-cd348fbc835c-kube-api-access-66trh\") pod \"node-resolver-8thwh\" (UID: \"d462c3b4-ffb3-4449-aff4-cd348fbc835c\") " pod="openshift-dns/node-resolver-8thwh" Jul 02 20:14:53.593302 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593193 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-var-lib-cni-multus\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.593302 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593227 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn22x\" (UniqueName: \"kubernetes.io/projected/0b7cafcc-8972-4475-8f86-4e22f41d1604-kube-api-access-bn22x\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.593302 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593268 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-cnibin\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.593302 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593287 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-kubernetes\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.593302 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593303 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-systemd\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.593601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593328 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjbm\" (UniqueName: \"kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm\") pod \"network-check-target-764hb\" (UID: \"57d801c5-6014-4d74-a071-b8ead03d72d1\") " pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:14:53.593601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593350 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b7cafcc-8972-4475-8f86-4e22f41d1604-cni-binary-copy\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.593601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593368 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa235a05-22fe-4e3a-adc5-705913aad0c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.593601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593396 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-run-ovn\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.593601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593420 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-node-log\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.593601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593447 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6797\" (UniqueName: \"kubernetes.io/projected/fa235a05-22fe-4e3a-adc5-705913aad0c4-kube-api-access-x6797\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.593601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593473 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.593601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593498 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f0c99d5-da5d-434c-99fe-762eff122a1e-ovnkube-script-lib\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.593601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593522 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dbc6243-cea2-42c1-88da-d3c44128342e-tmp\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.593601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-system-cni-dir\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.593601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593577 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-systemd-units\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.593962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593666 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-run-netns\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.593962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-log-socket\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.593962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-cni-netd\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.593962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593701 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-cnibin\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.593962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:53.593962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-kubelet\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.593962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593774 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.593962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593836 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-var-lib-cni-bin\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.593962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593860 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-kubernetes\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.593962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593909 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-var-lib-cni-bin\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.593962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593932 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-systemd\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.594345 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594197 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-daemon-config\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.594345 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.594345 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594274 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-slash\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.594345 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-lib-modules\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.594551 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594465 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-lib-modules\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.594551 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594484 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.594551 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594511 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-slash\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.594551 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-run-systemd\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.594719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594553 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa235a05-22fe-4e3a-adc5-705913aad0c4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.594719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594570 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0b7cafcc-8972-4475-8f86-4e22f41d1604-cni-binary-copy\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.594719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594574 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f0c99d5-da5d-434c-99fe-762eff122a1e-env-overrides\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.594719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594610 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-systemd-units\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.594719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594619 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-dbus\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:53.594719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594652 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f0c99d5-da5d-434c-99fe-762eff122a1e-ovnkube-config\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.594719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594677 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-sys\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.594719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594707 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-cni-dir\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.594968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594731 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-os-release\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.594968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594753 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-run-ovn\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.594968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594758 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-run-k8s-cni-cncf-io\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.594968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594786 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-etc-openvswitch\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.594968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594810 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-node-log\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.594968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fgg8\" (UniqueName: \"kubernetes.io/projected/2f0c99d5-da5d-434c-99fe-762eff122a1e-kube-api-access-8fgg8\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.594968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594864 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-sysconfig\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.594968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594890 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-host\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.594968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-run-multus-certs\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.594968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594940 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-system-cni-dir\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.594968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594965 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-cni-bin\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594978 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2f0c99d5-da5d-434c-99fe-762eff122a1e-env-overrides\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.594990 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d462c3b4-ffb3-4449-aff4-cd348fbc835c-hosts-file\") pod \"node-resolver-8thwh\" (UID: \"d462c3b4-ffb3-4449-aff4-cd348fbc835c\") " pod="openshift-dns/node-resolver-8thwh" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595034 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-sysctl-conf\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-socket-dir-parent\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595072 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-var-lib-kubelet\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595120 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/d462c3b4-ffb3-4449-aff4-cd348fbc835c-hosts-file\") pod \"node-resolver-8thwh\" (UID: \"d462c3b4-ffb3-4449-aff4-cd348fbc835c\") " pod="openshift-dns/node-resolver-8thwh" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595135 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-hostroot\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595153 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-daemon-config\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595168 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-dbus\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595214 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-run-systemd\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595228 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-run-netns\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595234 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.595301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595299 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-log-socket\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595305 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-cni-bin\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595332 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-os-release\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595382 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-hostroot\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.595386 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595425 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-socket-dir-parent\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595457 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-var-lib-kubelet\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-conf-dir\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595532 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-modprobe-d\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.595578 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret podName:81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee nodeName:}" failed. No retries permitted until 2026-07-02 20:14:54.095557131 +0000 UTC m=+3.091860344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret") pod "global-pull-secret-syncer-nkzfc" (UID: "81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.593772 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-var-lib-cni-multus\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595621 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-system-cni-dir\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595621 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh95n\" (UniqueName: \"kubernetes.io/projected/6dbc6243-cea2-42c1-88da-d3c44128342e-kube-api-access-gh95n\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-os-release\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595696 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-run\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595723 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-tuned\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595751 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-etc-kubernetes\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.595913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595776 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-sysctl-d\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-var-lib-kubelet\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595831 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa235a05-22fe-4e3a-adc5-705913aad0c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595861 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-sysconfig\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595863 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-var-lib-openvswitch\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595868 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2f0c99d5-da5d-434c-99fe-762eff122a1e-ovnkube-script-lib\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595903 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-var-lib-openvswitch\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595908 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-run-openvswitch\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595946 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2f0c99d5-da5d-434c-99fe-762eff122a1e-ovnkube-config\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595983 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-run-multus-certs\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.595967 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-cni-netd\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596032 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-sysctl-d\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596031 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-kubelet\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596051 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa235a05-22fe-4e3a-adc5-705913aad0c4-os-release\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596075 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-run\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596091 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596128 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-host\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.596811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596147 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-sysctl-conf\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596150 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-kubelet-config\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596176 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-kubelet-config\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa235a05-22fe-4e3a-adc5-705913aad0c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596199 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-run-openvswitch\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596220 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d462c3b4-ffb3-4449-aff4-cd348fbc835c-tmp-dir\") pod \"node-resolver-8thwh\" (UID: \"d462c3b4-ffb3-4449-aff4-cd348fbc835c\") " pod="openshift-dns/node-resolver-8thwh" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596244 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-conf-dir\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596253 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-cnibin\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596265 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-run-k8s-cni-cncf-io\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-run-netns\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596327 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-system-cni-dir\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596366 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-multus-cni-dir\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596392 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa235a05-22fe-4e3a-adc5-705913aad0c4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596412 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-sys\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596453 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-var-lib-kubelet\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596497 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-etc-kubernetes\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596554 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-cnibin\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596624 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d462c3b4-ffb3-4449-aff4-cd348fbc835c-tmp-dir\") pod \"node-resolver-8thwh\" (UID: \"d462c3b4-ffb3-4449-aff4-cd348fbc835c\") " pod="openshift-dns/node-resolver-8thwh" Jul 02 20:14:53.597751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596741 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-modprobe-d\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.598668 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596876 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b7cafcc-8972-4475-8f86-4e22f41d1604-host-run-netns\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.598668 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.596899 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2f0c99d5-da5d-434c-99fe-762eff122a1e-etc-openvswitch\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.598668 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.597068 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa235a05-22fe-4e3a-adc5-705913aad0c4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.598668 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.597336 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2f0c99d5-da5d-434c-99fe-762eff122a1e-ovn-node-metrics-cert\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.598668 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.598573 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6dbc6243-cea2-42c1-88da-d3c44128342e-etc-tuned\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.598985 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.598964 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6dbc6243-cea2-42c1-88da-d3c44128342e-tmp\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.603720 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.603695 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66trh\" (UniqueName: \"kubernetes.io/projected/d462c3b4-ffb3-4449-aff4-cd348fbc835c-kube-api-access-66trh\") pod \"node-resolver-8thwh\" (UID: \"d462c3b4-ffb3-4449-aff4-cd348fbc835c\") " pod="openshift-dns/node-resolver-8thwh" Jul 02 20:14:53.604554 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.604518 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn22x\" (UniqueName: \"kubernetes.io/projected/0b7cafcc-8972-4475-8f86-4e22f41d1604-kube-api-access-bn22x\") pod \"multus-nswz8\" (UID: \"0b7cafcc-8972-4475-8f86-4e22f41d1604\") " pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.606874 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.606849 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fgg8\" (UniqueName: \"kubernetes.io/projected/2f0c99d5-da5d-434c-99fe-762eff122a1e-kube-api-access-8fgg8\") pod \"ovnkube-node-h6ssd\" (UID: \"2f0c99d5-da5d-434c-99fe-762eff122a1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.607200 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.607180 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6797\" (UniqueName: \"kubernetes.io/projected/fa235a05-22fe-4e3a-adc5-705913aad0c4-kube-api-access-x6797\") pod \"multus-additional-cni-plugins-5c7nw\" (UID: \"fa235a05-22fe-4e3a-adc5-705913aad0c4\") " pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.607573 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.607553 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh95n\" (UniqueName: \"kubernetes.io/projected/6dbc6243-cea2-42c1-88da-d3c44128342e-kube-api-access-gh95n\") pod \"tuned-g8mkj\" (UID: \"6dbc6243-cea2-42c1-88da-d3c44128342e\") " pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.610541 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.610495 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:14:53.610541 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.610518 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:14:53.610541 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.610533 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gvjbm for pod openshift-network-diagnostics/network-check-target-764hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:53.610716 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.610601 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm podName:57d801c5-6014-4d74-a071-b8ead03d72d1 nodeName:}" failed. No retries permitted until 2026-07-02 20:14:54.110582458 +0000 UTC m=+3.106885691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gvjbm" (UniqueName: "kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm") pod "network-check-target-764hb" (UID: "57d801c5-6014-4d74-a071-b8ead03d72d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:53.687821 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.687780 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" Jul 02 20:14:53.699514 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.699486 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jz4g9" Jul 02 20:14:53.712579 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.712558 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" Jul 02 20:14:53.720364 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.720346 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-shnxp" Jul 02 20:14:53.729389 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.729368 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:14:53.750245 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.750225 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" Jul 02 20:14:53.757377 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.757351 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nswz8" Jul 02 20:14:53.765293 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.765273 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:14:53.766258 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.766241 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8thwh" Jul 02 20:14:53.868573 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.868491 2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 20:14:53.998993 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:53.998947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:53.999173 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.999097 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:53.999173 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:53.999173 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs podName:8caf8e2d-384f-4ecb-a60c-3ce8137a419a nodeName:}" failed. No retries permitted until 2026-07-02 20:14:54.999151008 +0000 UTC m=+3.995454223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs") pod "network-metrics-daemon-hr888" (UID: "8caf8e2d-384f-4ecb-a60c-3ce8137a419a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:54.099465 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:54.099423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:54.099645 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:54.099594 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:54.099716 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:54.099668 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret podName:81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee nodeName:}" failed. No retries permitted until 2026-07-02 20:14:55.099649173 +0000 UTC m=+4.095952385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret") pod "global-pull-secret-syncer-nkzfc" (UID: "81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:54.200729 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:54.200628 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjbm\" (UniqueName: \"kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm\") pod \"network-check-target-764hb\" (UID: \"57d801c5-6014-4d74-a071-b8ead03d72d1\") " pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:14:54.200908 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:54.200847 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:14:54.200908 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:54.200872 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:14:54.200908 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:54.200886 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gvjbm for pod openshift-network-diagnostics/network-check-target-764hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:54.201080 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:54.200949 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm podName:57d801c5-6014-4d74-a071-b8ead03d72d1 nodeName:}" failed. No retries permitted until 2026-07-02 20:14:55.200933069 +0000 UTC m=+4.197236285 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvjbm" (UniqueName: "kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm") pod "network-check-target-764hb" (UID: "57d801c5-6014-4d74-a071-b8ead03d72d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:54.426395 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:54.426352 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 20:09:53 +0000 UTC" deadline="2028-03-25 01:23:09.87695081 +0000 UTC" Jul 02 20:14:54.426395 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:54.426388 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15149h8m15.450568654s" Jul 02 20:14:54.565612 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:54.565516 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:54.565774 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:54.565661 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:14:54.681361 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:54.681317 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podded82f6b_f240_4500_9e22_bcc20618b526.slice/crio-1946e171d48f491a095543babbc414cdd046f94dfa6b5149579629f3f224ee25 WatchSource:0}: Error finding container 1946e171d48f491a095543babbc414cdd046f94dfa6b5149579629f3f224ee25: Status 404 returned error can't find the container with id 1946e171d48f491a095543babbc414cdd046f94dfa6b5149579629f3f224ee25 Jul 02 20:14:54.682478 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:54.682455 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7cafcc_8972_4475_8f86_4e22f41d1604.slice/crio-395fd85762c93e1cdc27a08782d5fa114b9fa38d9172a3ed34c6e303bbeb8c03 WatchSource:0}: Error finding container 395fd85762c93e1cdc27a08782d5fa114b9fa38d9172a3ed34c6e303bbeb8c03: Status 404 returned error can't find the container with id 395fd85762c93e1cdc27a08782d5fa114b9fa38d9172a3ed34c6e303bbeb8c03 Jul 02 20:14:54.682729 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:54.682705 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbc6243_cea2_42c1_88da_d3c44128342e.slice/crio-db6fdf0bf641e0f1d4ced99a35102b6133c07c5352178fab3b49813847393ce3 WatchSource:0}: Error finding container db6fdf0bf641e0f1d4ced99a35102b6133c07c5352178fab3b49813847393ce3: Status 404 returned error can't find the container with id db6fdf0bf641e0f1d4ced99a35102b6133c07c5352178fab3b49813847393ce3 Jul 02 20:14:54.683831 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:54.683807 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aae5d08_0b74_4611_8932_d1afc7b80412.slice/crio-6af24f87ccda4090903b878cf8d0cf35693814d06ef3ec7ca22df0390d6afb1e WatchSource:0}: Error finding container 6af24f87ccda4090903b878cf8d0cf35693814d06ef3ec7ca22df0390d6afb1e: Status 404 returned error can't find the container with id 6af24f87ccda4090903b878cf8d0cf35693814d06ef3ec7ca22df0390d6afb1e Jul 02 20:14:54.685964 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:54.685940 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd462c3b4_ffb3_4449_aff4_cd348fbc835c.slice/crio-8ce9e601f9ff6eaa63241a5ec41a7691f8922470acf244b207cb6ad91969166b WatchSource:0}: Error finding container 8ce9e601f9ff6eaa63241a5ec41a7691f8922470acf244b207cb6ad91969166b: Status 404 returned error can't find the container with id 8ce9e601f9ff6eaa63241a5ec41a7691f8922470acf244b207cb6ad91969166b Jul 02 20:14:54.691758 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:54.691600 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f0c99d5_da5d_434c_99fe_762eff122a1e.slice/crio-82aab6ad225f989ddabe21c3ed8f65ef72315c0f13102536a3acf16e678530a8 WatchSource:0}: Error finding container 82aab6ad225f989ddabe21c3ed8f65ef72315c0f13102536a3acf16e678530a8: Status 404 returned error can't find the container with id 82aab6ad225f989ddabe21c3ed8f65ef72315c0f13102536a3acf16e678530a8 Jul 02 20:14:54.696981 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:14:54.696260 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9ef4988_0ad9_491c_9935_929c5a0eb8ba.slice/crio-5e3ff062855b231362be105d0b2e6599677c39564c59c112e145c71737c9d03c WatchSource:0}: Error finding container 5e3ff062855b231362be105d0b2e6599677c39564c59c112e145c71737c9d03c: Status 404 returned error can't find the container with id 5e3ff062855b231362be105d0b2e6599677c39564c59c112e145c71737c9d03c Jul 02 20:14:55.008543 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.008513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:55.008690 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:55.008647 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:55.008752 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:55.008699 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs podName:8caf8e2d-384f-4ecb-a60c-3ce8137a419a nodeName:}" failed. No retries permitted until 2026-07-02 20:14:57.008684959 +0000 UTC m=+6.004988174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs") pod "network-metrics-daemon-hr888" (UID: "8caf8e2d-384f-4ecb-a60c-3ce8137a419a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:55.109732 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.109691 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:55.109905 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:55.109830 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:55.109905 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:55.109884 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret podName:81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee nodeName:}" failed. No retries permitted until 2026-07-02 20:14:57.109870117 +0000 UTC m=+6.106173330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret") pod "global-pull-secret-syncer-nkzfc" (UID: "81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:55.211041 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.210953 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjbm\" (UniqueName: \"kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm\") pod \"network-check-target-764hb\" (UID: \"57d801c5-6014-4d74-a071-b8ead03d72d1\") " pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:14:55.211222 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:55.211159 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:14:55.211222 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:55.211188 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:14:55.211222 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:55.211201 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gvjbm for pod openshift-network-diagnostics/network-check-target-764hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:55.211417 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:55.211278 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm podName:57d801c5-6014-4d74-a071-b8ead03d72d1 nodeName:}" failed. No retries permitted until 2026-07-02 20:14:57.21124207 +0000 UTC m=+6.207545290 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvjbm" (UniqueName: "kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm") pod "network-check-target-764hb" (UID: "57d801c5-6014-4d74-a071-b8ead03d72d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:55.427540 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.427469 2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 20:09:53 +0000 UTC" deadline="2028-01-30 01:39:02.033798178 +0000 UTC" Jul 02 20:14:55.427540 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.427537 2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13829h24m6.606265163s" Jul 02 20:14:55.565459 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.565384 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:55.565627 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:55.565527 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:14:55.566082 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.566060 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:14:55.566220 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:55.566162 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:14:55.588300 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.588264 2565 generic.go:356] "Generic (PLEG): container finished" podID="70902e0777cf94797571da24b10ed602" containerID="976d3c77d749a3c6c347f2bfc5bec16e120bd9dc8409f74e660dfba52b6b1e96" exitCode=0 Jul 02 20:14:55.588974 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.588521 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" event={"ID":"70902e0777cf94797571da24b10ed602","Type":"ContainerDied","Data":"976d3c77d749a3c6c347f2bfc5bec16e120bd9dc8409f74e660dfba52b6b1e96"} Jul 02 20:14:55.599185 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.599124 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jz4g9" event={"ID":"a9ef4988-0ad9-491c-9935-929c5a0eb8ba","Type":"ContainerStarted","Data":"5e3ff062855b231362be105d0b2e6599677c39564c59c112e145c71737c9d03c"} Jul 02 20:14:55.601497 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.601469 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" event={"ID":"fa235a05-22fe-4e3a-adc5-705913aad0c4","Type":"ContainerStarted","Data":"4657abb0b74608550842fd44cf071efd929f26a8d04f74f5cb5bf6547df6e316"} Jul 02 20:14:55.610099 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.610012 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-shnxp" event={"ID":"8180e463-918f-413b-ae8f-5ceabe81e0fe","Type":"ContainerStarted","Data":"65ba8cf3d71ba14c3bfc9b4ae2d14aa56a3c065b2477ac836e12b72d51ad4f71"} Jul 02 20:14:55.615690 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.615663 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hd7jg" event={"ID":"3aae5d08-0b74-4611-8932-d1afc7b80412","Type":"ContainerStarted","Data":"6af24f87ccda4090903b878cf8d0cf35693814d06ef3ec7ca22df0390d6afb1e"} Jul 02 20:14:55.634343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.631162 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" event={"ID":"6dbc6243-cea2-42c1-88da-d3c44128342e","Type":"ContainerStarted","Data":"db6fdf0bf641e0f1d4ced99a35102b6133c07c5352178fab3b49813847393ce3"} Jul 02 20:14:55.654836 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.654797 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" event={"ID":"ded82f6b-f240-4500-9e22-bcc20618b526","Type":"ContainerStarted","Data":"1946e171d48f491a095543babbc414cdd046f94dfa6b5149579629f3f224ee25"} Jul 02 20:14:55.669999 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.669884 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" event={"ID":"2f0c99d5-da5d-434c-99fe-762eff122a1e","Type":"ContainerStarted","Data":"82aab6ad225f989ddabe21c3ed8f65ef72315c0f13102536a3acf16e678530a8"} Jul 02 20:14:55.690687 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.690599 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8thwh" event={"ID":"d462c3b4-ffb3-4449-aff4-cd348fbc835c","Type":"ContainerStarted","Data":"8ce9e601f9ff6eaa63241a5ec41a7691f8922470acf244b207cb6ad91969166b"} Jul 02 20:14:55.713589 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.713515 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nswz8" event={"ID":"0b7cafcc-8972-4475-8f86-4e22f41d1604","Type":"ContainerStarted","Data":"395fd85762c93e1cdc27a08782d5fa114b9fa38d9172a3ed34c6e303bbeb8c03"} Jul 02 20:14:55.719591 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:55.718944 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-15.ec2.internal" event={"ID":"99ae426d085798cd8010de0c176c4766","Type":"ContainerStarted","Data":"a4ff4d9aad584bb1a6daf1acf101e265cb150f0503b16eadef77c300d1f09006"} Jul 02 20:14:56.565450 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:56.565416 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:56.565946 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:56.565567 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:14:56.745695 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:56.745655 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" event={"ID":"70902e0777cf94797571da24b10ed602","Type":"ContainerStarted","Data":"aa4618a76d589cb79be4c6cac1f10923d2099f42f1bb37921700e6a29cff81b6"} Jul 02 20:14:56.761855 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:56.761793 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-15.ec2.internal" podStartSLOduration=3.761769597 podStartE2EDuration="3.761769597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:14:56.761504604 +0000 UTC m=+5.757807841" watchObservedRunningTime="2026-07-02 20:14:56.761769597 +0000 UTC m=+5.758072938" Jul 02 20:14:56.762376 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:56.762310 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-15.ec2.internal" podStartSLOduration=3.7622729059999998 podStartE2EDuration="3.762272906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:14:55.736592226 +0000 UTC m=+4.732895464" watchObservedRunningTime="2026-07-02 20:14:56.762272906 +0000 UTC m=+5.758576145" Jul 02 20:14:57.028581 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:57.028489 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:57.028750 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:57.028668 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:57.028750 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:57.028738 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs podName:8caf8e2d-384f-4ecb-a60c-3ce8137a419a nodeName:}" failed. No retries permitted until 2026-07-02 20:15:01.028718842 +0000 UTC m=+10.025022060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs") pod "network-metrics-daemon-hr888" (UID: "8caf8e2d-384f-4ecb-a60c-3ce8137a419a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:14:57.129594 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:57.129547 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:57.129767 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:57.129701 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:57.129767 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:57.129766 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret podName:81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee nodeName:}" failed. No retries permitted until 2026-07-02 20:15:01.129748602 +0000 UTC m=+10.126051833 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret") pod "global-pull-secret-syncer-nkzfc" (UID: "81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:14:57.231564 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:57.230849 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjbm\" (UniqueName: \"kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm\") pod \"network-check-target-764hb\" (UID: \"57d801c5-6014-4d74-a071-b8ead03d72d1\") " pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:14:57.231564 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:57.231075 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:14:57.231564 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:57.231098 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:14:57.231564 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:57.231114 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gvjbm for pod openshift-network-diagnostics/network-check-target-764hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:57.231564 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:57.231179 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm podName:57d801c5-6014-4d74-a071-b8ead03d72d1 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:01.231158185 +0000 UTC m=+10.227461403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvjbm" (UniqueName: "kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm") pod "network-check-target-764hb" (UID: "57d801c5-6014-4d74-a071-b8ead03d72d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:14:57.565769 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:57.565729 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:14:57.566267 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:57.565857 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:14:57.567992 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:57.565696 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:57.567992 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:57.567464 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:14:58.565970 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:58.565441 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:14:58.565970 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:58.565591 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:14:59.570602 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:59.569929 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:14:59.570602 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:59.570055 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:14:59.570602 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:14:59.569938 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:14:59.570602 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:14:59.570504 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:00.565603 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:00.565067 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:00.565603 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:00.565235 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:01.064275 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:01.064187 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:01.064718 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:01.064329 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:01.064718 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:01.064380 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs podName:8caf8e2d-384f-4ecb-a60c-3ce8137a419a nodeName:}" failed. No retries permitted until 2026-07-02 20:15:09.064367408 +0000 UTC m=+18.060670620 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs") pod "network-metrics-daemon-hr888" (UID: "8caf8e2d-384f-4ecb-a60c-3ce8137a419a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:01.164721 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:01.164686 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:01.164879 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:01.164840 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:01.164938 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:01.164914 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret podName:81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee nodeName:}" failed. No retries permitted until 2026-07-02 20:15:09.164893601 +0000 UTC m=+18.161196813 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret") pod "global-pull-secret-syncer-nkzfc" (UID: "81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:01.265174 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:01.265137 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjbm\" (UniqueName: \"kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm\") pod \"network-check-target-764hb\" (UID: \"57d801c5-6014-4d74-a071-b8ead03d72d1\") " pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:01.265391 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:01.265354 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:15:01.265391 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:01.265383 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:15:01.265562 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:01.265398 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gvjbm for pod openshift-network-diagnostics/network-check-target-764hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:01.265562 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:01.265460 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm podName:57d801c5-6014-4d74-a071-b8ead03d72d1 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:09.26544094 +0000 UTC m=+18.261744155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvjbm" (UniqueName: "kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm") pod "network-check-target-764hb" (UID: "57d801c5-6014-4d74-a071-b8ead03d72d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:01.565982 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:01.565950 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:01.566133 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:01.566035 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:01.566343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:01.566330 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:01.566403 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:01.566394 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:02.565307 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:02.565275 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:02.565721 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:02.565389 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:03.565731 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:03.565650 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:03.566156 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:03.565776 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:03.566156 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:03.565824 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:03.566156 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:03.565920 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:04.565671 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:04.565643 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:04.565859 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:04.565761 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:05.565326 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:05.565289 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:05.565494 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:05.565443 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:05.565555 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:05.565490 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:05.565612 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:05.565590 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:06.565156 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:06.565120 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:06.565604 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:06.565302 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:07.565066 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:07.565032 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:07.565270 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:07.565037 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:07.565270 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:07.565153 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:07.565734 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:07.565263 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:08.565718 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:08.565684 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:08.566244 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:08.565808 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:09.125317 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:09.125282 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:09.125490 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:09.125441 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:09.125559 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:09.125528 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs podName:8caf8e2d-384f-4ecb-a60c-3ce8137a419a nodeName:}" failed. No retries permitted until 2026-07-02 20:15:25.125499797 +0000 UTC m=+34.121803015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs") pod "network-metrics-daemon-hr888" (UID: "8caf8e2d-384f-4ecb-a60c-3ce8137a419a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:09.226995 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:09.226431 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:09.226995 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:09.226558 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:09.226995 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:09.226642 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret podName:81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee nodeName:}" failed. No retries permitted until 2026-07-02 20:15:25.226619893 +0000 UTC m=+34.222923124 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret") pod "global-pull-secret-syncer-nkzfc" (UID: "81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:09.327156 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:09.327119 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjbm\" (UniqueName: \"kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm\") pod \"network-check-target-764hb\" (UID: \"57d801c5-6014-4d74-a071-b8ead03d72d1\") " pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:09.327366 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:09.327288 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:15:09.327366 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:09.327313 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:15:09.327366 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:09.327326 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gvjbm for pod openshift-network-diagnostics/network-check-target-764hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:09.327514 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:09.327391 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm podName:57d801c5-6014-4d74-a071-b8ead03d72d1 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:25.327373107 +0000 UTC m=+34.323676339 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvjbm" (UniqueName: "kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm") pod "network-check-target-764hb" (UID: "57d801c5-6014-4d74-a071-b8ead03d72d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:09.565135 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:09.565061 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:09.565311 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:09.565189 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:09.565311 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:09.565268 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:09.565421 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:09.565376 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:10.565169 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:10.565138 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:10.565636 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:10.565277 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:11.566693 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:11.566652 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:11.567134 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:11.566762 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:11.567134 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:11.566839 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:11.567134 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:11.566953 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:12.566007 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:12.565968 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:12.566192 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:12.566122 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:13.565922 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:13.565307 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:13.565922 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:13.565438 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:13.565922 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:13.565790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:13.565922 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:13.565879 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:14.566129 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.565713 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:14.566710 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:14.566198 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:14.791980 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.791894 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hd7jg" event={"ID":"3aae5d08-0b74-4611-8932-d1afc7b80412","Type":"ContainerStarted","Data":"70860c17d634013e3dd2aed07d86bfae9f0a6580a3db64e151c21442e4f6e35c"} Jul 02 20:15:14.793363 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.793340 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" event={"ID":"6dbc6243-cea2-42c1-88da-d3c44128342e","Type":"ContainerStarted","Data":"e300fb30621fd222446a9e8f2bc6e0b67834920b69eeb8773150e05590d42a7b"} Jul 02 20:15:14.794825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.794793 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" event={"ID":"ded82f6b-f240-4500-9e22-bcc20618b526","Type":"ContainerStarted","Data":"df0af53c3e0fec816bf5ab010faca996a9e762c1c9a8addb3099eb87ba480895"} Jul 02 20:15:14.798150 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.798128 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" event={"ID":"2f0c99d5-da5d-434c-99fe-762eff122a1e","Type":"ContainerStarted","Data":"d5ba57d9d2baeaa16f5945ab249721737557e8862ea38474380c1124982618df"} Jul 02 20:15:14.798274 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.798185 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" event={"ID":"2f0c99d5-da5d-434c-99fe-762eff122a1e","Type":"ContainerStarted","Data":"dca5a9422b20eda00e7f9e289cda61aa1d2888a6cc7e4c02cef33e787bf72d14"} Jul 02 20:15:14.798274 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.798240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" event={"ID":"2f0c99d5-da5d-434c-99fe-762eff122a1e","Type":"ContainerStarted","Data":"d4c155ee964e15582a543ab5eccc4cd9f1d5917534bc8f4389b7e66abd0d4a6f"} Jul 02 20:15:14.798360 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.798285 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" event={"ID":"2f0c99d5-da5d-434c-99fe-762eff122a1e","Type":"ContainerStarted","Data":"a3bfdb0b183f7a7a8b596437ebc70f69116028e2bfc77ffd90854e71e7b07cf9"} Jul 02 20:15:14.798360 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.798313 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" event={"ID":"2f0c99d5-da5d-434c-99fe-762eff122a1e","Type":"ContainerStarted","Data":"07dfae2e34165ecc6d005dc824b3299f97f4a29c62db3fd0493c6badf33cc060"} Jul 02 20:15:14.798360 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.798339 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" event={"ID":"2f0c99d5-da5d-434c-99fe-762eff122a1e","Type":"ContainerStarted","Data":"1b19d111f665c3c594ac8468169d8280d4f128efac8416f76d1f3badd056a839"} Jul 02 20:15:14.799520 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.799500 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8thwh" event={"ID":"d462c3b4-ffb3-4449-aff4-cd348fbc835c","Type":"ContainerStarted","Data":"b637a76b6a1bef2b43a16a052718258eae408ae7c580b93654547e3993bdd04b"} Jul 02 20:15:14.800910 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.800882 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nswz8" event={"ID":"0b7cafcc-8972-4475-8f86-4e22f41d1604","Type":"ContainerStarted","Data":"16870ac3999a12d0df54f730a958ae7870f595e31d0daac903555bfdfb3f5af2"} Jul 02 20:15:14.802198 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.802179 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jz4g9" event={"ID":"a9ef4988-0ad9-491c-9935-929c5a0eb8ba","Type":"ContainerStarted","Data":"4b7d0e2abefcc6db518d7ad5e1544728eb76e02330716ac5df8219460c1b6d97"} Jul 02 20:15:14.803930 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.803905 2565 generic.go:356] "Generic (PLEG): container finished" podID="fa235a05-22fe-4e3a-adc5-705913aad0c4" containerID="51cf6294a413a4daad007761098d283d434bf2df06dc3cf9e220c856cee4e716" exitCode=0 Jul 02 20:15:14.804036 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.803946 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" event={"ID":"fa235a05-22fe-4e3a-adc5-705913aad0c4","Type":"ContainerDied","Data":"51cf6294a413a4daad007761098d283d434bf2df06dc3cf9e220c856cee4e716"} Jul 02 20:15:14.809257 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.809198 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hd7jg" podStartSLOduration=11.629042064 podStartE2EDuration="22.809181848s" totalImagesPullingTime="11.180139784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:14.808622819 +0000 UTC m=+23.804926054" watchObservedRunningTime="2026-07-02 20:15:14.809181848 +0000 UTC m=+23.805485086" Jul 02 20:15:14.826795 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.826751 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-g8mkj" podStartSLOduration=3.607614736 podStartE2EDuration="22.826735512s" totalImagesPullingTime="19.219120776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:14.826425062 +0000 UTC m=+23.822728296" watchObservedRunningTime="2026-07-02 20:15:14.826735512 +0000 UTC m=+23.823038744" Jul 02 20:15:14.880255 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.880157 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8thwh" podStartSLOduration=3.667136775 podStartE2EDuration="22.880140097s" totalImagesPullingTime="19.213003322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:14.880060338 +0000 UTC m=+23.876363573" watchObservedRunningTime="2026-07-02 20:15:14.880140097 +0000 UTC m=+23.876443330" Jul 02 20:15:14.880480 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.880457 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nswz8" podStartSLOduration=3.604018228 podStartE2EDuration="22.880447746s" totalImagesPullingTime="19.276429518s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:14.86615627 +0000 UTC m=+23.862459494" watchObservedRunningTime="2026-07-02 20:15:14.880447746 +0000 UTC m=+23.876750979" Jul 02 20:15:14.895012 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:14.894973 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jz4g9" podStartSLOduration=3.691190478 podStartE2EDuration="22.894960679s" totalImagesPullingTime="19.203770201s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:14.894829886 +0000 UTC m=+23.891133121" watchObservedRunningTime="2026-07-02 20:15:14.894960679 +0000 UTC m=+23.891263911" Jul 02 20:15:15.009700 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.009666 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:15.085183 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.085107 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:15.342970 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.342892 2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 20:15:15.460393 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.460259 2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T20:15:15.342912986Z","UUID":"98055264-3e4c-438d-aa93-6670991ced22","Handler":null,"Name":"","Endpoint":""} Jul 02 20:15:15.461725 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.461710 2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 20:15:15.461811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.461746 2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 20:15:15.565902 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.565867 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:15.566134 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:15.566002 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:15.566134 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.566051 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:15.566501 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:15.566135 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:15.730452 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.730376 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:15.808101 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.808066 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-shnxp" event={"ID":"8180e463-918f-413b-ae8f-5ceabe81e0fe","Type":"ContainerStarted","Data":"f2a63f30745c2067df1fc310ca688f24a0b745727ccaf8091141f57d3b012309"} Jul 02 20:15:15.811270 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.810698 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" event={"ID":"ded82f6b-f240-4500-9e22-bcc20618b526","Type":"ContainerStarted","Data":"b09029db0e6356128c798aecdb69ba9b024197950480ccb58cd368dc30ee6b82"} Jul 02 20:15:15.823336 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.823283 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-shnxp" podStartSLOduration=4.610363407 podStartE2EDuration="23.823262073s" totalImagesPullingTime="19.212898666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:15.822580014 +0000 UTC m=+24.818883248" watchObservedRunningTime="2026-07-02 20:15:15.823262073 +0000 UTC m=+24.819565309" Jul 02 20:15:15.826790 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.826767 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:15.955111 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:15.955072 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:16.005004 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:16.004925 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:16.539225 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:16.539183 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:16.565109 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:16.565086 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:16.565281 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:16.565187 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:16.746318 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:16.746292 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:16.815221 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:16.815181 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" event={"ID":"ded82f6b-f240-4500-9e22-bcc20618b526","Type":"ContainerStarted","Data":"8c903ecc10d2971f86614005a7e1814d9f7ba58fd5e7730fbc5d5f2f3caf25f6"} Jul 02 20:15:16.818539 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:16.818505 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" event={"ID":"2f0c99d5-da5d-434c-99fe-762eff122a1e","Type":"ContainerStarted","Data":"49b4143197bf41be1211acea1e0834d171a9c70bacef6a78d73f175a9b707252"} Jul 02 20:15:16.832144 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:16.832093 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-9d78t" podStartSLOduration=3.5308912279999998 podStartE2EDuration="24.832073467s" totalImagesPullingTime="21.301182239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:16.832025664 +0000 UTC m=+25.828328921" watchObservedRunningTime="2026-07-02 20:15:16.832073467 +0000 UTC m=+25.828376697" Jul 02 20:15:17.145955 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:17.145882 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:17.246752 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:17.246721 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:17.347015 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:17.346984 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:15:17.347541 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:17.347526 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:15:17.565377 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:17.565298 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:17.565547 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:17.565302 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:17.565547 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:17.565416 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:17.565547 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:17.565524 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:17.821575 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:17.821494 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:15:17.821994 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:17.821984 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hd7jg" Jul 02 20:15:17.889883 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:17.889698 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:17.990063 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:17.990006 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:18.342402 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:18.342367 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:18.443420 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:18.443393 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:18.566016 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:18.565986 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:18.566257 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:18.566166 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:19.565134 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:19.565053 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:19.565134 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:19.565099 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:19.565883 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:19.565179 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:19.565883 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:19.565278 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:20.565749 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:20.565727 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:20.566194 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:20.565836 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:20.834537 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:20.834352 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" event={"ID":"2f0c99d5-da5d-434c-99fe-762eff122a1e","Type":"ContainerStarted","Data":"ccc6aabcb360726071986fb503455228c2d5ceac228ef66f10102961a99e295a"} Jul 02 20:15:20.835243 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:20.834838 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:15:20.835243 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:20.835089 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:15:20.835243 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:20.835137 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:15:20.837394 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:20.837359 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" event={"ID":"fa235a05-22fe-4e3a-adc5-705913aad0c4","Type":"ContainerStarted","Data":"bbe0623c1cbe479a0790d0e83b37d91c14006885bdbd0b6d61968831ee716c39"} Jul 02 20:15:20.851411 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:20.851388 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:15:20.851520 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:20.851482 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:15:20.861647 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:20.861603 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" podStartSLOduration=9.592524616 podStartE2EDuration="28.861590196s" totalImagesPullingTime="19.26906558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:20.85993973 +0000 UTC m=+29.856242965" watchObservedRunningTime="2026-07-02 20:15:20.861590196 +0000 UTC m=+29.857893445" Jul 02 20:15:21.045658 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:21.045591 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:21.099595 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:21.099566 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:21.566111 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:21.566075 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:21.566704 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:21.566176 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:21.566704 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:21.566228 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:21.566704 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:21.566322 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:21.841137 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:21.841059 2565 generic.go:356] "Generic (PLEG): container finished" podID="fa235a05-22fe-4e3a-adc5-705913aad0c4" containerID="bbe0623c1cbe479a0790d0e83b37d91c14006885bdbd0b6d61968831ee716c39" exitCode=0 Jul 02 20:15:21.841291 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:21.841148 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" event={"ID":"fa235a05-22fe-4e3a-adc5-705913aad0c4","Type":"ContainerDied","Data":"bbe0623c1cbe479a0790d0e83b37d91c14006885bdbd0b6d61968831ee716c39"} Jul 02 20:15:22.565587 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:22.565556 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:22.565760 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:22.565691 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:22.645826 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:22.645794 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nkzfc"] Jul 02 20:15:22.646172 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:22.645907 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:22.646172 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:22.645991 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:22.649090 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:22.649067 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-764hb"] Jul 02 20:15:22.649251 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:22.649158 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:22.649315 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:22.649272 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:22.649714 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:22.649690 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hr888"] Jul 02 20:15:22.843698 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:22.843674 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:22.843842 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:22.843789 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:23.847732 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:23.847446 2565 generic.go:356] "Generic (PLEG): container finished" podID="fa235a05-22fe-4e3a-adc5-705913aad0c4" containerID="d9cca05b013ce6f7c603e0537bf10e14c0531cc66f52d3123b6a95dccdfe5b05" exitCode=0 Jul 02 20:15:23.847732 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:23.847529 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" event={"ID":"fa235a05-22fe-4e3a-adc5-705913aad0c4","Type":"ContainerDied","Data":"d9cca05b013ce6f7c603e0537bf10e14c0531cc66f52d3123b6a95dccdfe5b05"} Jul 02 20:15:24.565443 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:24.565406 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:24.565597 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:24.565414 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:24.565597 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:24.565539 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:24.565597 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:24.565559 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:24.565748 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:24.565669 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:24.565796 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:24.565767 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:25.142217 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:25.142177 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:25.142551 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:25.142287 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:25.142551 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:25.142347 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs podName:8caf8e2d-384f-4ecb-a60c-3ce8137a419a nodeName:}" failed. No retries permitted until 2026-07-02 20:15:57.142331448 +0000 UTC m=+66.138634659 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs") pod "network-metrics-daemon-hr888" (UID: "8caf8e2d-384f-4ecb-a60c-3ce8137a419a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 20:15:25.242917 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:25.242891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:25.243031 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:25.243010 2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:25.243074 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:25.243058 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret podName:81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee nodeName:}" failed. No retries permitted until 2026-07-02 20:15:57.243044482 +0000 UTC m=+66.239347694 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret") pod "global-pull-secret-syncer-nkzfc" (UID: "81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee") : object "kube-system"/"original-pull-secret" not registered Jul 02 20:15:25.343385 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:25.343297 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjbm\" (UniqueName: \"kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm\") pod \"network-check-target-764hb\" (UID: \"57d801c5-6014-4d74-a071-b8ead03d72d1\") " pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:25.343526 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:25.343414 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 20:15:25.343526 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:25.343428 2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 20:15:25.343526 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:25.343438 2565 projected.go:196] Error preparing data for projected volume kube-api-access-gvjbm for pod openshift-network-diagnostics/network-check-target-764hb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:25.343526 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:25.343484 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm podName:57d801c5-6014-4d74-a071-b8ead03d72d1 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:57.343470687 +0000 UTC m=+66.339773897 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gvjbm" (UniqueName: "kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm") pod "network-check-target-764hb" (UID: "57d801c5-6014-4d74-a071-b8ead03d72d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 20:15:25.854282 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:25.854252 2565 generic.go:356] "Generic (PLEG): container finished" podID="fa235a05-22fe-4e3a-adc5-705913aad0c4" containerID="df4384c30ef1791630afe718f2e1418ccaa7c6d4bfe5d148c8108778a3299220" exitCode=0 Jul 02 20:15:25.854444 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:25.854304 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" event={"ID":"fa235a05-22fe-4e3a-adc5-705913aad0c4","Type":"ContainerDied","Data":"df4384c30ef1791630afe718f2e1418ccaa7c6d4bfe5d148c8108778a3299220"} Jul 02 20:15:26.565549 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:26.565516 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:26.566129 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:26.565639 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nkzfc" podUID="81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee" Jul 02 20:15:26.566129 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:26.566059 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:26.566261 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:26.566158 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:15:26.566261 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:26.566222 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:26.566371 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:26.566292 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-764hb" podUID="57d801c5-6014-4d74-a071-b8ead03d72d1" Jul 02 20:15:27.813138 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.813060 2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-15.ec2.internal" event="NodeReady" Jul 02 20:15:27.813563 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.813192 2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 20:15:27.854980 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.854947 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hmr7z"] Jul 02 20:15:27.871800 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.870898 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gj22w"] Jul 02 20:15:27.871800 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.871019 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:27.874129 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.874100 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 20:15:27.874129 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.874112 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 20:15:27.874330 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.874156 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzb4r\"" Jul 02 20:15:27.882756 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.882735 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmr7z"] Jul 02 20:15:27.882861 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.882777 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gj22w"] Jul 02 20:15:27.882861 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.882854 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gj22w" Jul 02 20:15:27.885426 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.885402 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 20:15:27.885528 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.885428 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 20:15:27.885528 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.885451 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 20:15:27.890885 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.890862 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7swmr\"" Jul 02 20:15:27.931353 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.931320 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:27.961405 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.961368 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:27.961576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.961409 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-config-volume\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:27.961576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.961507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-tmp-dir\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:27.961576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:27.961549 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c6np\" (UniqueName: \"kubernetes.io/projected/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-kube-api-access-7c6np\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:28.034534 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.034502 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:28.062548 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.062512 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-config-volume\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:28.062548 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.062551 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5rf2\" (UniqueName: \"kubernetes.io/projected/efc20391-221b-452a-8960-bd8d55818dc9-kube-api-access-m5rf2\") pod \"ingress-canary-gj22w\" (UID: \"efc20391-221b-452a-8960-bd8d55818dc9\") " pod="openshift-ingress-canary/ingress-canary-gj22w" Jul 02 20:15:28.062790 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.062588 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-tmp-dir\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:28.062790 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.062734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c6np\" (UniqueName: \"kubernetes.io/projected/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-kube-api-access-7c6np\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:28.062900 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.062805 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:28.062900 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.062832 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efc20391-221b-452a-8960-bd8d55818dc9-cert\") pod \"ingress-canary-gj22w\" (UID: \"efc20391-221b-452a-8960-bd8d55818dc9\") " pod="openshift-ingress-canary/ingress-canary-gj22w" Jul 02 20:15:28.062999 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.062981 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-tmp-dir\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:28.063085 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:28.063062 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:28.063305 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:28.063136 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls podName:3386b537-14c7-4ed3-9c7c-7d5d83e7ae93 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:28.563116427 +0000 UTC m=+37.559419641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls") pod "dns-default-hmr7z" (UID: "3386b537-14c7-4ed3-9c7c-7d5d83e7ae93") : secret "dns-default-metrics-tls" not found Jul 02 20:15:28.063305 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.063275 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-config-volume\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:28.074492 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.074462 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c6np\" (UniqueName: \"kubernetes.io/projected/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-kube-api-access-7c6np\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:28.163460 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.163423 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efc20391-221b-452a-8960-bd8d55818dc9-cert\") pod \"ingress-canary-gj22w\" (UID: \"efc20391-221b-452a-8960-bd8d55818dc9\") " pod="openshift-ingress-canary/ingress-canary-gj22w" Jul 02 20:15:28.163460 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.163459 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5rf2\" (UniqueName: \"kubernetes.io/projected/efc20391-221b-452a-8960-bd8d55818dc9-kube-api-access-m5rf2\") pod \"ingress-canary-gj22w\" (UID: \"efc20391-221b-452a-8960-bd8d55818dc9\") " pod="openshift-ingress-canary/ingress-canary-gj22w" Jul 02 20:15:28.165960 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.165940 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/efc20391-221b-452a-8960-bd8d55818dc9-cert\") pod \"ingress-canary-gj22w\" (UID: \"efc20391-221b-452a-8960-bd8d55818dc9\") " pod="openshift-ingress-canary/ingress-canary-gj22w" Jul 02 20:15:28.172779 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.172757 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5rf2\" (UniqueName: \"kubernetes.io/projected/efc20391-221b-452a-8960-bd8d55818dc9-kube-api-access-m5rf2\") pod \"ingress-canary-gj22w\" (UID: \"efc20391-221b-452a-8960-bd8d55818dc9\") " pod="openshift-ingress-canary/ingress-canary-gj22w" Jul 02 20:15:28.192697 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.192667 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gj22w" Jul 02 20:15:28.307795 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.307769 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:28.386574 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.386537 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gj22w"] Jul 02 20:15:28.389946 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.389925 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:28.392039 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:15:28.392016 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc20391_221b_452a_8960_bd8d55818dc9.slice/crio-b94bcab719ba9e72d11e4b9b994838ff6f1b41d92951bf6dd093fc7604cdaac9 WatchSource:0}: Error finding container b94bcab719ba9e72d11e4b9b994838ff6f1b41d92951bf6dd093fc7604cdaac9: Status 404 returned error can't find the container with id b94bcab719ba9e72d11e4b9b994838ff6f1b41d92951bf6dd093fc7604cdaac9 Jul 02 20:15:28.565396 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.565363 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:28.565587 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.565369 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:28.565587 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.565369 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:28.566112 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.566092 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:28.566260 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:28.566246 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:28.566325 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:28.566315 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls podName:3386b537-14c7-4ed3-9c7c-7d5d83e7ae93 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:29.566293775 +0000 UTC m=+38.562596988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls") pod "dns-default-hmr7z" (UID: "3386b537-14c7-4ed3-9c7c-7d5d83e7ae93") : secret "dns-default-metrics-tls" not found Jul 02 20:15:28.568555 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.568536 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 20:15:28.568671 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.568606 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 20:15:28.568671 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.568629 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8kznm\"" Jul 02 20:15:28.568671 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.568649 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 20:15:28.568671 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.568648 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2jpl6\"" Jul 02 20:15:28.569573 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.569555 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 20:15:28.862515 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:28.862454 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gj22w" event={"ID":"efc20391-221b-452a-8960-bd8d55818dc9","Type":"ContainerStarted","Data":"b94bcab719ba9e72d11e4b9b994838ff6f1b41d92951bf6dd093fc7604cdaac9"} Jul 02 20:15:29.572511 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:29.572478 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:29.572681 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:29.572623 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:29.572681 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:29.572675 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls podName:3386b537-14c7-4ed3-9c7c-7d5d83e7ae93 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:31.572660644 +0000 UTC m=+40.568963854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls") pod "dns-default-hmr7z" (UID: "3386b537-14c7-4ed3-9c7c-7d5d83e7ae93") : secret "dns-default-metrics-tls" not found Jul 02 20:15:31.538627 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:31.538595 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:31.590853 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:31.590785 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:31.590994 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:31.590926 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:31.591048 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:31.591022 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls podName:3386b537-14c7-4ed3-9c7c-7d5d83e7ae93 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:35.591006074 +0000 UTC m=+44.587309288 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls") pod "dns-default-hmr7z" (UID: "3386b537-14c7-4ed3-9c7c-7d5d83e7ae93") : secret "dns-default-metrics-tls" not found Jul 02 20:15:32.273848 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:32.273613 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:32.339182 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:32.339137 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:32.874102 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:32.874068 2565 generic.go:356] "Generic (PLEG): container finished" podID="fa235a05-22fe-4e3a-adc5-705913aad0c4" containerID="bf89d462d329c50be710df8b67e6c962164c82c8d0abbbb8a5636fd3fc39281a" exitCode=0 Jul 02 20:15:32.874545 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:32.874107 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" event={"ID":"fa235a05-22fe-4e3a-adc5-705913aad0c4","Type":"ContainerDied","Data":"bf89d462d329c50be710df8b67e6c962164c82c8d0abbbb8a5636fd3fc39281a"} Jul 02 20:15:32.875616 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:32.875595 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gj22w" event={"ID":"efc20391-221b-452a-8960-bd8d55818dc9","Type":"ContainerStarted","Data":"17d21b40644e01f0c8fc51120d7ccfbe23b203f0a18bd7f64d073218117501d6"} Jul 02 20:15:32.909544 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:32.909503 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gj22w" podStartSLOduration=2.338768935 podStartE2EDuration="5.909489458s" totalImagesPullingTime="3.570720523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:32.908363259 +0000 UTC m=+41.904666493" watchObservedRunningTime="2026-07-02 20:15:32.909489458 +0000 UTC m=+41.905792669" Jul 02 20:15:33.697407 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:33.697368 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:33.778107 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:33.778076 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:33.881171 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:33.881144 2565 generic.go:356] "Generic (PLEG): container finished" podID="fa235a05-22fe-4e3a-adc5-705913aad0c4" containerID="09a2bb21416cc376be9ca44a93e90e8dda36c63ec5a22046a9b9ca655bb98950" exitCode=0 Jul 02 20:15:33.881560 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:33.881240 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" event={"ID":"fa235a05-22fe-4e3a-adc5-705913aad0c4","Type":"ContainerDied","Data":"09a2bb21416cc376be9ca44a93e90e8dda36c63ec5a22046a9b9ca655bb98950"} Jul 02 20:15:34.690382 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:34.690344 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:34.752613 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:34.752581 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:34.886628 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:34.886596 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" event={"ID":"fa235a05-22fe-4e3a-adc5-705913aad0c4","Type":"ContainerStarted","Data":"ebede874ca9aa6e5e48eadd3d09b4a4f8db4fa6ce3ac2568f120a461916d2783"} Jul 02 20:15:34.909801 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:34.909751 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5c7nw" podStartSLOduration=9.520910961 podStartE2EDuration="42.909735244s" totalImagesPullingTime="33.388824283s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:15:34.907740222 +0000 UTC m=+43.904043457" watchObservedRunningTime="2026-07-02 20:15:34.909735244 +0000 UTC m=+43.906038476" Jul 02 20:15:35.036988 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:35.036909 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:35.092294 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:35.092263 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:35.621919 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:35.621879 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:35.622068 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:35.622026 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:35.622128 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:35.622097 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls podName:3386b537-14c7-4ed3-9c7c-7d5d83e7ae93 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:43.622081315 +0000 UTC m=+52.618384527 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls") pod "dns-default-hmr7z" (UID: "3386b537-14c7-4ed3-9c7c-7d5d83e7ae93") : secret "dns-default-metrics-tls" not found Jul 02 20:15:43.675922 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:43.675883 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:43.676470 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:43.675997 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:43.676470 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:43.676047 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls podName:3386b537-14c7-4ed3-9c7c-7d5d83e7ae93 nodeName:}" failed. No retries permitted until 2026-07-02 20:15:59.676032435 +0000 UTC m=+68.672335645 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls") pod "dns-default-hmr7z" (UID: "3386b537-14c7-4ed3-9c7c-7d5d83e7ae93") : secret "dns-default-metrics-tls" not found Jul 02 20:15:52.859171 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:52.859144 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6ssd" Jul 02 20:15:53.006433 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:53.006405 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:53.064613 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:53.064584 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:54.620255 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:54.620221 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:54.674300 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:54.674272 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:54.728410 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:54.728373 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:15:57.169269 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.169237 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:15:57.172366 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.172346 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 20:15:57.179578 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:57.179558 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 02 20:15:57.179656 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:57.179630 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs podName:8caf8e2d-384f-4ecb-a60c-3ce8137a419a nodeName:}" failed. No retries permitted until 2026-07-02 20:17:01.179609551 +0000 UTC m=+130.175912762 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs") pod "network-metrics-daemon-hr888" (UID: "8caf8e2d-384f-4ecb-a60c-3ce8137a419a") : secret "metrics-daemon-secret" not found Jul 02 20:15:57.270343 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.270307 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:57.273135 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.273117 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 20:15:57.284879 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.284855 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee-original-pull-secret\") pod \"global-pull-secret-syncer-nkzfc\" (UID: \"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee\") " pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:57.370601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.370576 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvjbm\" (UniqueName: \"kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm\") pod \"network-check-target-764hb\" (UID: \"57d801c5-6014-4d74-a071-b8ead03d72d1\") " pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:57.373610 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.373592 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 20:15:57.375705 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.375691 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nkzfc" Jul 02 20:15:57.384035 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.384004 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 20:15:57.395073 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.395044 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvjbm\" (UniqueName: \"kubernetes.io/projected/57d801c5-6014-4d74-a071-b8ead03d72d1-kube-api-access-gvjbm\") pod \"network-check-target-764hb\" (UID: \"57d801c5-6014-4d74-a071-b8ead03d72d1\") " pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:57.498329 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.498299 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nkzfc"] Jul 02 20:15:57.501014 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:15:57.500985 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81dfeaf2_e4c7_4ac8_8ec3_7b4bb8ed5eee.slice/crio-0357b9dc9fb376fa3c02fb9071d4e2391b5671821bbc7e61760455c9d677fece WatchSource:0}: Error finding container 0357b9dc9fb376fa3c02fb9071d4e2391b5671821bbc7e61760455c9d677fece: Status 404 returned error can't find the container with id 0357b9dc9fb376fa3c02fb9071d4e2391b5671821bbc7e61760455c9d677fece Jul 02 20:15:57.684100 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.684023 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-8kznm\"" Jul 02 20:15:57.691978 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.691953 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:15:57.809658 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.809634 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-764hb"] Jul 02 20:15:57.811961 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:15:57.811933 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57d801c5_6014_4d74_a071_b8ead03d72d1.slice/crio-e965a0dae2678a3d7ea3a9f397e802da204cf984d5f7d07754dbcf3e25bb27b8 WatchSource:0}: Error finding container e965a0dae2678a3d7ea3a9f397e802da204cf984d5f7d07754dbcf3e25bb27b8: Status 404 returned error can't find the container with id e965a0dae2678a3d7ea3a9f397e802da204cf984d5f7d07754dbcf3e25bb27b8 Jul 02 20:15:57.937475 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.937395 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-764hb" event={"ID":"57d801c5-6014-4d74-a071-b8ead03d72d1","Type":"ContainerStarted","Data":"e965a0dae2678a3d7ea3a9f397e802da204cf984d5f7d07754dbcf3e25bb27b8"} Jul 02 20:15:57.938418 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:57.938397 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nkzfc" event={"ID":"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee","Type":"ContainerStarted","Data":"0357b9dc9fb376fa3c02fb9071d4e2391b5671821bbc7e61760455c9d677fece"} Jul 02 20:15:59.688139 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:15:59.688100 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:15:59.688573 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:59.688255 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:15:59.688573 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:15:59.688333 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls podName:3386b537-14c7-4ed3-9c7c-7d5d83e7ae93 nodeName:}" failed. No retries permitted until 2026-07-02 20:16:31.688314148 +0000 UTC m=+100.684617365 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls") pod "dns-default-hmr7z" (UID: "3386b537-14c7-4ed3-9c7c-7d5d83e7ae93") : secret "dns-default-metrics-tls" not found Jul 02 20:16:02.953486 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:02.953451 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-764hb" event={"ID":"57d801c5-6014-4d74-a071-b8ead03d72d1","Type":"ContainerStarted","Data":"9e262ff8c68cf0121ea8657f2d020281707bbb1608e3d154bc6c4b668084f80f"} Jul 02 20:16:02.953939 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:02.953566 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:16:02.954918 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:02.954899 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nkzfc" event={"ID":"81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee","Type":"ContainerStarted","Data":"7eabe7287fc0d776a6b09c44bf60ca331fdfee1263efe16aa7957336de54fd54"} Jul 02 20:16:02.969962 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:02.969918 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-764hb" podStartSLOduration=66.329981186 podStartE2EDuration="1m10.96990383s" totalImagesPullingTime="4.639922644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:16:02.968782719 +0000 UTC m=+71.965085952" watchObservedRunningTime="2026-07-02 20:16:02.96990383 +0000 UTC m=+71.966207106" Jul 02 20:16:02.982785 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:02.982744 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nkzfc" podStartSLOduration=66.029791356 podStartE2EDuration="1m10.982729302s" totalImagesPullingTime="4.952937946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:16:02.98226558 +0000 UTC m=+71.978568813" watchObservedRunningTime="2026-07-02 20:16:02.982729302 +0000 UTC m=+71.979032572" Jul 02 20:16:07.267247 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.267192 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw"] Jul 02 20:16:07.299329 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.299300 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw"] Jul 02 20:16:07.299458 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.299390 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:07.301972 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.301952 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 02 20:16:07.303289 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.303269 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 02 20:16:07.303378 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.303330 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 02 20:16:07.303378 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.303361 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 02 20:16:07.350914 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.350890 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27652715-6ed1-4e6c-8af4-ac870a04c70b-tmp\") pod \"klusterlet-addon-workmgr-7d5fff7d8c-7b6nw\" (UID: \"27652715-6ed1-4e6c-8af4-ac870a04c70b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:07.351023 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.350915 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tm2w\" (UniqueName: \"kubernetes.io/projected/27652715-6ed1-4e6c-8af4-ac870a04c70b-kube-api-access-4tm2w\") pod \"klusterlet-addon-workmgr-7d5fff7d8c-7b6nw\" (UID: \"27652715-6ed1-4e6c-8af4-ac870a04c70b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:07.351023 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.350948 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/27652715-6ed1-4e6c-8af4-ac870a04c70b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d5fff7d8c-7b6nw\" (UID: \"27652715-6ed1-4e6c-8af4-ac870a04c70b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:07.452375 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.452342 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/27652715-6ed1-4e6c-8af4-ac870a04c70b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d5fff7d8c-7b6nw\" (UID: \"27652715-6ed1-4e6c-8af4-ac870a04c70b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:07.452530 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.452427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27652715-6ed1-4e6c-8af4-ac870a04c70b-tmp\") pod \"klusterlet-addon-workmgr-7d5fff7d8c-7b6nw\" (UID: \"27652715-6ed1-4e6c-8af4-ac870a04c70b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:07.452530 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.452446 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tm2w\" (UniqueName: \"kubernetes.io/projected/27652715-6ed1-4e6c-8af4-ac870a04c70b-kube-api-access-4tm2w\") pod \"klusterlet-addon-workmgr-7d5fff7d8c-7b6nw\" (UID: \"27652715-6ed1-4e6c-8af4-ac870a04c70b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:07.452890 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.452868 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27652715-6ed1-4e6c-8af4-ac870a04c70b-tmp\") pod \"klusterlet-addon-workmgr-7d5fff7d8c-7b6nw\" (UID: \"27652715-6ed1-4e6c-8af4-ac870a04c70b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:07.455070 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.455050 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/27652715-6ed1-4e6c-8af4-ac870a04c70b-klusterlet-config\") pod \"klusterlet-addon-workmgr-7d5fff7d8c-7b6nw\" (UID: \"27652715-6ed1-4e6c-8af4-ac870a04c70b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:07.460349 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.460329 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tm2w\" (UniqueName: \"kubernetes.io/projected/27652715-6ed1-4e6c-8af4-ac870a04c70b-kube-api-access-4tm2w\") pod \"klusterlet-addon-workmgr-7d5fff7d8c-7b6nw\" (UID: \"27652715-6ed1-4e6c-8af4-ac870a04c70b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:07.608086 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.607999 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:07.724242 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.724193 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw"] Jul 02 20:16:07.727260 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:16:07.727231 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27652715_6ed1_4e6c_8af4_ac870a04c70b.slice/crio-9a6b05153819fcb0cf0b6a6004b84e6777fb524a281686ad3cd5d1fa974595f2 WatchSource:0}: Error finding container 9a6b05153819fcb0cf0b6a6004b84e6777fb524a281686ad3cd5d1fa974595f2: Status 404 returned error can't find the container with id 9a6b05153819fcb0cf0b6a6004b84e6777fb524a281686ad3cd5d1fa974595f2 Jul 02 20:16:07.967872 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:07.967836 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" event={"ID":"27652715-6ed1-4e6c-8af4-ac870a04c70b","Type":"ContainerStarted","Data":"9a6b05153819fcb0cf0b6a6004b84e6777fb524a281686ad3cd5d1fa974595f2"} Jul 02 20:16:11.979822 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:11.979783 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" event={"ID":"27652715-6ed1-4e6c-8af4-ac870a04c70b","Type":"ContainerStarted","Data":"853991bb04e0a38ce61878c05a2a5067fdd69b0e75b32901e6c250205ef41de9"} Jul 02 20:16:11.980237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:11.980089 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:11.981562 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:11.981540 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:16:12.010616 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:12.010576 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" podStartSLOduration=1.146923798 podStartE2EDuration="5.01056148s" totalImagesPullingTime="3.863637682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:16:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:16:11.994872222 +0000 UTC m=+80.991175457" watchObservedRunningTime="2026-07-02 20:16:12.01056148 +0000 UTC m=+81.006864712" Jul 02 20:16:19.116030 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:19.116002 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:16:31.719429 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:31.719385 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:16:31.719796 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:16:31.719540 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:16:31.719796 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:16:31.719608 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls podName:3386b537-14c7-4ed3-9c7c-7d5d83e7ae93 nodeName:}" failed. No retries permitted until 2026-07-02 20:17:35.719589494 +0000 UTC m=+164.715892706 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls") pod "dns-default-hmr7z" (UID: "3386b537-14c7-4ed3-9c7c-7d5d83e7ae93") : secret "dns-default-metrics-tls" not found Jul 02 20:16:33.960363 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:16:33.960329 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-764hb" Jul 02 20:17:01.226146 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:17:01.226104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:17:01.226667 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:17:01.226252 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 02 20:17:01.226667 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:17:01.226333 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs podName:8caf8e2d-384f-4ecb-a60c-3ce8137a419a nodeName:}" failed. No retries permitted until 2026-07-02 20:19:03.226317343 +0000 UTC m=+252.222620554 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs") pod "network-metrics-daemon-hr888" (UID: "8caf8e2d-384f-4ecb-a60c-3ce8137a419a") : secret "metrics-daemon-secret" not found Jul 02 20:17:30.882389 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:17:30.882353 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hmr7z" podUID="3386b537-14c7-4ed3-9c7c-7d5d83e7ae93" Jul 02 20:17:31.165454 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:17:31.165430 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmr7z" Jul 02 20:17:31.586660 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:17:31.586584 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:17:35.759419 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:17:35.759375 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:17:35.759871 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:17:35.759519 2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 20:17:35.759871 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:17:35.759590 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls podName:3386b537-14c7-4ed3-9c7c-7d5d83e7ae93 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:37.759571263 +0000 UTC m=+286.755874482 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls") pod "dns-default-hmr7z" (UID: "3386b537-14c7-4ed3-9c7c-7d5d83e7ae93") : secret "dns-default-metrics-tls" not found Jul 02 20:17:43.565791 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:17:43.565711 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:18:11.980619 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:11.980504 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" podUID="27652715-6ed1-4e6c-8af4-ac870a04c70b" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.8:8000/readyz\": dial tcp 10.133.0.8:8000: connect: connection refused" Jul 02 20:18:12.263698 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:12.263612 2565 generic.go:356] "Generic (PLEG): container finished" podID="27652715-6ed1-4e6c-8af4-ac870a04c70b" containerID="853991bb04e0a38ce61878c05a2a5067fdd69b0e75b32901e6c250205ef41de9" exitCode=1 Jul 02 20:18:12.263829 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:12.263690 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" event={"ID":"27652715-6ed1-4e6c-8af4-ac870a04c70b","Type":"ContainerDied","Data":"853991bb04e0a38ce61878c05a2a5067fdd69b0e75b32901e6c250205ef41de9"} Jul 02 20:18:12.264024 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:12.264011 2565 scope.go:117] "RemoveContainer" containerID="853991bb04e0a38ce61878c05a2a5067fdd69b0e75b32901e6c250205ef41de9" Jul 02 20:18:13.267904 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:13.267873 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" event={"ID":"27652715-6ed1-4e6c-8af4-ac870a04c70b","Type":"ContainerStarted","Data":"9b3eaafcc376daa2448b069723f5be3c9f2c938546c71934c8cfc0b923dc16ad"} Jul 02 20:18:13.268288 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:13.268140 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:18:13.268722 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:13.268705 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7d5fff7d8c-7b6nw" Jul 02 20:18:14.806007 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:14.805981 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:48.748365 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.748329 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67"] Jul 02 20:18:48.751468 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.751452 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k"] Jul 02 20:18:48.751581 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.751563 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:18:48.756183 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.756149 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b5cf7cccf-9vswd"] Jul 02 20:18:48.756341 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.756188 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 02 20:18:48.756341 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.756197 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 02 20:18:48.756341 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.756231 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 02 20:18:48.756341 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.756198 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:48.756341 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.756274 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q9zqq\"" Jul 02 20:18:48.758337 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.758319 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-7b8w7"] Jul 02 20:18:48.758456 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.758440 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.759223 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.759189 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j5mmh\"" Jul 02 20:18:48.759309 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.759226 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 02 20:18:48.759309 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.759228 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 20:18:48.759415 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.759309 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 20:18:48.760351 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.760089 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-7r8wz"] Jul 02 20:18:48.760351 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.760320 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.760895 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.760877 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 02 20:18:48.761960 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.761946 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.763936 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.763920 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 02 20:18:48.765879 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.765859 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 02 20:18:48.765968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.765876 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 20:18:48.765968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.765906 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 02 20:18:48.766131 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.766118 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 02 20:18:48.766391 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.766295 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 02 20:18:48.766391 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.766308 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7jgs\"" Jul 02 20:18:48.766391 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.766342 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 02 20:18:48.766391 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.766369 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 02 20:18:48.766602 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.766413 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-zrvg9\"" Jul 02 20:18:48.766602 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.766297 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 02 20:18:48.766602 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.766437 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 20:18:48.767084 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.767064 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-5fd44\"" Jul 02 20:18:48.767249 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.767085 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 02 20:18:48.774380 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.774362 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k"] Jul 02 20:18:48.775496 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.775431 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67"] Jul 02 20:18:48.776340 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.776320 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b5cf7cccf-9vswd"] Jul 02 20:18:48.776648 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.776626 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 02 20:18:48.777194 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.777171 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 02 20:18:48.777376 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.777356 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-7r8wz"] Jul 02 20:18:48.777543 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.777525 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 02 20:18:48.786485 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.786464 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-7b8w7"] Jul 02 20:18:48.843714 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.843685 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-h6l8q"] Jul 02 20:18:48.845811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.845793 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h6l8q" Jul 02 20:18:48.848033 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e15797b-e921-4425-b53f-94b801477686-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.848124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848048 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e15797b-e921-4425-b53f-94b801477686-serving-cert\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.848124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848085 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-certificates\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.848253 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848130 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-installation-pull-secrets\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.848253 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848244 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w2mf\" (UniqueName: \"kubernetes.io/projected/6e15797b-e921-4425-b53f-94b801477686-kube-api-access-5w2mf\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.848456 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848267 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs9vz\" (UniqueName: \"kubernetes.io/projected/a7e7fea8-aaec-47ae-9137-5be0b93671bb-kube-api-access-rs9vz\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:48.848456 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848298 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-bound-sa-token\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.848456 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848316 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4gqt\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-kube-api-access-j4gqt\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.848456 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e15797b-e921-4425-b53f-94b801477686-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.848456 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848350 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trlcm\" (UniqueName: \"kubernetes.io/projected/cabf7e15-b90b-4363-939a-ac9f61cc07ac-kube-api-access-trlcm\") pod \"cluster-samples-operator-56fcf86bf-tdq67\" (UID: \"cabf7e15-b90b-4363-939a-ac9f61cc07ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:18:48.848456 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848365 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-ca-trust-extracted\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.848456 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ba6abb2-3ddd-4296-9858-6de372075f3a-config\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.848456 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848436 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.848770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848483 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmrm8\" (UniqueName: \"kubernetes.io/projected/5ba6abb2-3ddd-4296-9858-6de372075f3a-kube-api-access-qmrm8\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.848770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848500 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:48.848770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848550 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7vgql\"" Jul 02 20:18:48.848770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848550 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6abb2-3ddd-4296-9858-6de372075f3a-trusted-ca\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.848770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848588 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6e15797b-e921-4425-b53f-94b801477686-snapshots\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.848770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848612 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tdq67\" (UID: \"cabf7e15-b90b-4363-939a-ac9f61cc07ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:18:48.848770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e15797b-e921-4425-b53f-94b801477686-tmp\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.848770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a7e7fea8-aaec-47ae-9137-5be0b93671bb-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:48.848770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848719 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-trusted-ca\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.849077 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848773 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-image-registry-private-configuration\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.849077 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.848803 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ba6abb2-3ddd-4296-9858-6de372075f3a-serving-cert\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.862257 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.862240 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-h6l8q"] Jul 02 20:18:48.949242 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-image-registry-private-configuration\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.949242 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949243 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ba6abb2-3ddd-4296-9858-6de372075f3a-serving-cert\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.949476 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949269 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e15797b-e921-4425-b53f-94b801477686-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.949476 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949291 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e15797b-e921-4425-b53f-94b801477686-serving-cert\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.949476 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949334 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r942q\" (UniqueName: \"kubernetes.io/projected/f425063c-b422-4cd6-b71f-23aea5c99df8-kube-api-access-r942q\") pod \"network-check-source-989f8b8dc-h6l8q\" (UID: \"f425063c-b422-4cd6-b71f-23aea5c99df8\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h6l8q" Jul 02 20:18:48.949476 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949378 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-certificates\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.949476 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949400 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-installation-pull-secrets\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.949476 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949430 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w2mf\" (UniqueName: \"kubernetes.io/projected/6e15797b-e921-4425-b53f-94b801477686-kube-api-access-5w2mf\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.949476 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs9vz\" (UniqueName: \"kubernetes.io/projected/a7e7fea8-aaec-47ae-9137-5be0b93671bb-kube-api-access-rs9vz\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949486 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-bound-sa-token\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949516 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4gqt\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-kube-api-access-j4gqt\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e15797b-e921-4425-b53f-94b801477686-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949580 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trlcm\" (UniqueName: \"kubernetes.io/projected/cabf7e15-b90b-4363-939a-ac9f61cc07ac-kube-api-access-trlcm\") pod \"cluster-samples-operator-56fcf86bf-tdq67\" (UID: \"cabf7e15-b90b-4363-939a-ac9f61cc07ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949604 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-ca-trust-extracted\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949640 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ba6abb2-3ddd-4296-9858-6de372075f3a-config\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949665 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmrm8\" (UniqueName: \"kubernetes.io/projected/5ba6abb2-3ddd-4296-9858-6de372075f3a-kube-api-access-qmrm8\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949766 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6abb2-3ddd-4296-9858-6de372075f3a-trusted-ca\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949789 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6e15797b-e921-4425-b53f-94b801477686-snapshots\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.949825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949813 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tdq67\" (UID: \"cabf7e15-b90b-4363-939a-ac9f61cc07ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:18:48.950509 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e15797b-e921-4425-b53f-94b801477686-tmp\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.950509 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949873 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a7e7fea8-aaec-47ae-9137-5be0b93671bb-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:48.950509 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.949911 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-trusted-ca\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.950509 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.950070 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-ca-trust-extracted\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.950509 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.950070 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-certificates\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.950509 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.950351 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e15797b-e921-4425-b53f-94b801477686-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.950792 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:48.950770 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 20:18:48.950792 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:48.950787 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b5cf7cccf-9vswd: secret "image-registry-tls" not found Jul 02 20:18:48.950896 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:48.950839 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls podName:e20f83f3-3574-41f1-99c9-0e22df9a2e0d nodeName:}" failed. No retries permitted until 2026-07-02 20:18:49.450821375 +0000 UTC m=+238.447124590 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls") pod "image-registry-7b5cf7cccf-9vswd" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d") : secret "image-registry-tls" not found Jul 02 20:18:48.950896 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.950882 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e15797b-e921-4425-b53f-94b801477686-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.951002 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:48.950971 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 20:18:48.951054 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.951000 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-trusted-ca\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.951054 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:48.951032 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls podName:cabf7e15-b90b-4363-939a-ac9f61cc07ac nodeName:}" failed. No retries permitted until 2026-07-02 20:18:49.451014132 +0000 UTC m=+238.447317345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-tdq67" (UID: "cabf7e15-b90b-4363-939a-ac9f61cc07ac") : secret "samples-operator-tls" not found Jul 02 20:18:48.951251 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.951230 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ba6abb2-3ddd-4296-9858-6de372075f3a-trusted-ca\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.951398 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.951321 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e15797b-e921-4425-b53f-94b801477686-tmp\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.951617 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.951593 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ba6abb2-3ddd-4296-9858-6de372075f3a-config\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.951710 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.951629 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/6e15797b-e921-4425-b53f-94b801477686-snapshots\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.951710 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:48.951698 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:18:48.951832 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:48.951744 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls podName:a7e7fea8-aaec-47ae-9137-5be0b93671bb nodeName:}" failed. No retries permitted until 2026-07-02 20:18:49.451731093 +0000 UTC m=+238.448034305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mqr6k" (UID: "a7e7fea8-aaec-47ae-9137-5be0b93671bb") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:18:48.952203 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.952174 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a7e7fea8-aaec-47ae-9137-5be0b93671bb-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:48.952407 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.952385 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-installation-pull-secrets\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.952516 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.952496 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-image-registry-private-configuration\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.952553 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.952496 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ba6abb2-3ddd-4296-9858-6de372075f3a-serving-cert\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.953136 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.953113 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6e15797b-e921-4425-b53f-94b801477686-serving-cert\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.961662 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.961643 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmrm8\" (UniqueName: \"kubernetes.io/projected/5ba6abb2-3ddd-4296-9858-6de372075f3a-kube-api-access-qmrm8\") pod \"console-operator-68b757fb67-7b8w7\" (UID: \"5ba6abb2-3ddd-4296-9858-6de372075f3a\") " pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:48.962188 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.962168 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs9vz\" (UniqueName: \"kubernetes.io/projected/a7e7fea8-aaec-47ae-9137-5be0b93671bb-kube-api-access-rs9vz\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:48.962639 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.962617 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w2mf\" (UniqueName: \"kubernetes.io/projected/6e15797b-e921-4425-b53f-94b801477686-kube-api-access-5w2mf\") pod \"insights-operator-5c847dcfc4-7r8wz\" (UID: \"6e15797b-e921-4425-b53f-94b801477686\") " pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:48.962688 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.962670 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-bound-sa-token\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.962725 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.962699 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4gqt\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-kube-api-access-j4gqt\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:48.962765 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.962751 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trlcm\" (UniqueName: \"kubernetes.io/projected/cabf7e15-b90b-4363-939a-ac9f61cc07ac-kube-api-access-trlcm\") pod \"cluster-samples-operator-56fcf86bf-tdq67\" (UID: \"cabf7e15-b90b-4363-939a-ac9f61cc07ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:18:48.980994 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:48.980975 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:49.051333 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.051262 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r942q\" (UniqueName: \"kubernetes.io/projected/f425063c-b422-4cd6-b71f-23aea5c99df8-kube-api-access-r942q\") pod \"network-check-source-989f8b8dc-h6l8q\" (UID: \"f425063c-b422-4cd6-b71f-23aea5c99df8\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h6l8q" Jul 02 20:18:49.059673 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.059653 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r942q\" (UniqueName: \"kubernetes.io/projected/f425063c-b422-4cd6-b71f-23aea5c99df8-kube-api-access-r942q\") pod \"network-check-source-989f8b8dc-h6l8q\" (UID: \"f425063c-b422-4cd6-b71f-23aea5c99df8\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h6l8q" Jul 02 20:18:49.082386 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.082359 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:49.085298 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.085281 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:49.089945 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.089930 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" Jul 02 20:18:49.153766 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.153737 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h6l8q" Jul 02 20:18:49.216253 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.216174 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-7b8w7"] Jul 02 20:18:49.220435 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:18:49.220398 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ba6abb2_3ddd_4296_9858_6de372075f3a.slice/crio-6eb79f2548bb73fd97041c2293b9a44971a7e7f44323e680142c240506ad0990 WatchSource:0}: Error finding container 6eb79f2548bb73fd97041c2293b9a44971a7e7f44323e680142c240506ad0990: Status 404 returned error can't find the container with id 6eb79f2548bb73fd97041c2293b9a44971a7e7f44323e680142c240506ad0990 Jul 02 20:18:49.241591 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.241555 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-7r8wz"] Jul 02 20:18:49.244427 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:18:49.244404 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e15797b_e921_4425_b53f_94b801477686.slice/crio-f84a2bbbd4b6f2dca9599ffaf533d484644f44b5145907957ec2da4f0526585f WatchSource:0}: Error finding container f84a2bbbd4b6f2dca9599ffaf533d484644f44b5145907957ec2da4f0526585f: Status 404 returned error can't find the container with id f84a2bbbd4b6f2dca9599ffaf533d484644f44b5145907957ec2da4f0526585f Jul 02 20:18:49.293871 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.293738 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-h6l8q"] Jul 02 20:18:49.295956 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:18:49.295935 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf425063c_b422_4cd6_b71f_23aea5c99df8.slice/crio-38fbcd3837e956947a196327e096b7c859c3b9025befbdc42593ee18a4c2106d WatchSource:0}: Error finding container 38fbcd3837e956947a196327e096b7c859c3b9025befbdc42593ee18a4c2106d: Status 404 returned error can't find the container with id 38fbcd3837e956947a196327e096b7c859c3b9025befbdc42593ee18a4c2106d Jul 02 20:18:49.357976 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.357944 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" event={"ID":"6e15797b-e921-4425-b53f-94b801477686","Type":"ContainerStarted","Data":"f84a2bbbd4b6f2dca9599ffaf533d484644f44b5145907957ec2da4f0526585f"} Jul 02 20:18:49.359144 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.359114 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" event={"ID":"5ba6abb2-3ddd-4296-9858-6de372075f3a","Type":"ContainerStarted","Data":"6eb79f2548bb73fd97041c2293b9a44971a7e7f44323e680142c240506ad0990"} Jul 02 20:18:49.360588 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.360570 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h6l8q" event={"ID":"f425063c-b422-4cd6-b71f-23aea5c99df8","Type":"ContainerStarted","Data":"8cfb415fc4a73fc1798963c690ecf01db92e0085a018c84354fce15269e06d60"} Jul 02 20:18:49.360682 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.360601 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h6l8q" event={"ID":"f425063c-b422-4cd6-b71f-23aea5c99df8","Type":"ContainerStarted","Data":"38fbcd3837e956947a196327e096b7c859c3b9025befbdc42593ee18a4c2106d"} Jul 02 20:18:49.375955 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.375912 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-h6l8q" podStartSLOduration=1.375899757 podStartE2EDuration="1.375899757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:18:49.374992562 +0000 UTC m=+238.371295797" watchObservedRunningTime="2026-07-02 20:18:49.375899757 +0000 UTC m=+238.372203034" Jul 02 20:18:49.455566 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.455543 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:49.455696 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.455585 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:49.455696 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:49.455607 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tdq67\" (UID: \"cabf7e15-b90b-4363-939a-ac9f61cc07ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:18:49.455818 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:49.455718 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 20:18:49.455818 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:49.455725 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 20:18:49.455818 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:49.455735 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b5cf7cccf-9vswd: secret "image-registry-tls" not found Jul 02 20:18:49.455818 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:49.455740 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:18:49.455818 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:49.455784 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls podName:cabf7e15-b90b-4363-939a-ac9f61cc07ac nodeName:}" failed. No retries permitted until 2026-07-02 20:18:50.455766469 +0000 UTC m=+239.452069692 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-tdq67" (UID: "cabf7e15-b90b-4363-939a-ac9f61cc07ac") : secret "samples-operator-tls" not found Jul 02 20:18:49.455818 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:49.455797 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls podName:e20f83f3-3574-41f1-99c9-0e22df9a2e0d nodeName:}" failed. No retries permitted until 2026-07-02 20:18:50.455791324 +0000 UTC m=+239.452094535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls") pod "image-registry-7b5cf7cccf-9vswd" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d") : secret "image-registry-tls" not found Jul 02 20:18:49.455818 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:49.455807 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls podName:a7e7fea8-aaec-47ae-9137-5be0b93671bb nodeName:}" failed. No retries permitted until 2026-07-02 20:18:50.455801274 +0000 UTC m=+239.452104485 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mqr6k" (UID: "a7e7fea8-aaec-47ae-9137-5be0b93671bb") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:18:50.464673 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:50.464022 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:50.464673 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:50.464085 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:50.464673 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:50.464117 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tdq67\" (UID: \"cabf7e15-b90b-4363-939a-ac9f61cc07ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:18:50.464673 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:50.464134 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 20:18:50.464673 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:50.464149 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b5cf7cccf-9vswd: secret "image-registry-tls" not found Jul 02 20:18:50.464673 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:50.464204 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls podName:e20f83f3-3574-41f1-99c9-0e22df9a2e0d nodeName:}" failed. No retries permitted until 2026-07-02 20:18:52.464186858 +0000 UTC m=+241.460490086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls") pod "image-registry-7b5cf7cccf-9vswd" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d") : secret "image-registry-tls" not found Jul 02 20:18:50.464673 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:50.464583 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:18:50.464673 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:50.464621 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 20:18:50.464673 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:50.464633 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls podName:a7e7fea8-aaec-47ae-9137-5be0b93671bb nodeName:}" failed. No retries permitted until 2026-07-02 20:18:52.464617262 +0000 UTC m=+241.460920491 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mqr6k" (UID: "a7e7fea8-aaec-47ae-9137-5be0b93671bb") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:18:50.464673 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:50.464655 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls podName:cabf7e15-b90b-4363-939a-ac9f61cc07ac nodeName:}" failed. No retries permitted until 2026-07-02 20:18:52.464644516 +0000 UTC m=+241.460947731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-tdq67" (UID: "cabf7e15-b90b-4363-939a-ac9f61cc07ac") : secret "samples-operator-tls" not found Jul 02 20:18:51.280249 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:51.280192 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:51.386626 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:51.386587 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:52.372459 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:52.372376 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" event={"ID":"6e15797b-e921-4425-b53f-94b801477686","Type":"ContainerStarted","Data":"5988a3806dba30591f40243aab07df8a090b799ccd5739a9fd81b42824ed3bd7"} Jul 02 20:18:52.373995 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:52.373976 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/0.log" Jul 02 20:18:52.374119 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:52.374014 2565 generic.go:356] "Generic (PLEG): container finished" podID="5ba6abb2-3ddd-4296-9858-6de372075f3a" containerID="9c78abd756ec05a328376888e188e5c7dc8fd59b0ce7a3e8416a351e7fb1f1a9" exitCode=255 Jul 02 20:18:52.374119 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:52.374054 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" event={"ID":"5ba6abb2-3ddd-4296-9858-6de372075f3a","Type":"ContainerDied","Data":"9c78abd756ec05a328376888e188e5c7dc8fd59b0ce7a3e8416a351e7fb1f1a9"} Jul 02 20:18:52.374276 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:52.374263 2565 scope.go:117] "RemoveContainer" containerID="9c78abd756ec05a328376888e188e5c7dc8fd59b0ce7a3e8416a351e7fb1f1a9" Jul 02 20:18:52.392855 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:52.392805 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" podStartSLOduration=1.5158327470000001 podStartE2EDuration="4.392786722s" totalImagesPullingTime="2.876953975s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:18:52.39223502 +0000 UTC m=+241.388538253" watchObservedRunningTime="2026-07-02 20:18:52.392786722 +0000 UTC m=+241.389089956" Jul 02 20:18:52.481294 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:52.480767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:52.481294 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:52.480848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:52.481294 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:52.480902 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tdq67\" (UID: \"cabf7e15-b90b-4363-939a-ac9f61cc07ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:18:52.481294 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:52.481038 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 20:18:52.481294 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:52.481062 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 20:18:52.481294 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:52.481079 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b5cf7cccf-9vswd: secret "image-registry-tls" not found Jul 02 20:18:52.481294 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:52.481119 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls podName:cabf7e15-b90b-4363-939a-ac9f61cc07ac nodeName:}" failed. No retries permitted until 2026-07-02 20:18:56.481097882 +0000 UTC m=+245.477401094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-tdq67" (UID: "cabf7e15-b90b-4363-939a-ac9f61cc07ac") : secret "samples-operator-tls" not found Jul 02 20:18:52.481294 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:52.481139 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls podName:e20f83f3-3574-41f1-99c9-0e22df9a2e0d nodeName:}" failed. No retries permitted until 2026-07-02 20:18:56.481129057 +0000 UTC m=+245.477432268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls") pod "image-registry-7b5cf7cccf-9vswd" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d") : secret "image-registry-tls" not found Jul 02 20:18:52.481294 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:52.481170 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:18:52.481294 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:52.481238 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls podName:a7e7fea8-aaec-47ae-9137-5be0b93671bb nodeName:}" failed. No retries permitted until 2026-07-02 20:18:56.481195963 +0000 UTC m=+245.477499180 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mqr6k" (UID: "a7e7fea8-aaec-47ae-9137-5be0b93671bb") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:18:53.378450 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.378423 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/1.log" Jul 02 20:18:53.378873 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.378841 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/0.log" Jul 02 20:18:53.378916 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.378872 2565 generic.go:356] "Generic (PLEG): container finished" podID="5ba6abb2-3ddd-4296-9858-6de372075f3a" containerID="88be8e09b0c86c045f168630e6235668cd4ccba1ae8dad715c0eaeb35b62c708" exitCode=255 Jul 02 20:18:53.378979 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.378958 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" event={"ID":"5ba6abb2-3ddd-4296-9858-6de372075f3a","Type":"ContainerDied","Data":"88be8e09b0c86c045f168630e6235668cd4ccba1ae8dad715c0eaeb35b62c708"} Jul 02 20:18:53.379029 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.379010 2565 scope.go:117] "RemoveContainer" containerID="9c78abd756ec05a328376888e188e5c7dc8fd59b0ce7a3e8416a351e7fb1f1a9" Jul 02 20:18:53.379222 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.379178 2565 scope.go:117] "RemoveContainer" containerID="88be8e09b0c86c045f168630e6235668cd4ccba1ae8dad715c0eaeb35b62c708" Jul 02 20:18:53.379391 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:53.379374 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-7b8w7_openshift-console-operator(5ba6abb2-3ddd-4296-9858-6de372075f3a)\"" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" podUID="5ba6abb2-3ddd-4296-9858-6de372075f3a" Jul 02 20:18:53.693176 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.693147 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5wv7t"] Jul 02 20:18:53.696092 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.696075 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.699125 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.699106 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 20:18:53.699380 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.699365 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g7mm9\"" Jul 02 20:18:53.699576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.699562 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 20:18:53.707960 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.707941 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5wv7t"] Jul 02 20:18:53.719898 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.719877 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:53.774420 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.774386 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:53.789481 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.789456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.789558 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.789517 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbe0c436-b6d1-4773-9982-0d0323b30552-crio-socket\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.789623 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.789599 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe0c436-b6d1-4773-9982-0d0323b30552-data-volume\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.789662 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.789640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2m55\" (UniqueName: \"kubernetes.io/projected/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-api-access-c2m55\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.789695 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.789662 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.891019 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.890987 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.891129 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.891039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbe0c436-b6d1-4773-9982-0d0323b30552-crio-socket\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.891129 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.891066 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe0c436-b6d1-4773-9982-0d0323b30552-data-volume\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.891129 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.891084 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2m55\" (UniqueName: \"kubernetes.io/projected/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-api-access-c2m55\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.891129 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.891102 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.891129 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:53.891113 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:18:53.891332 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.891135 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dbe0c436-b6d1-4773-9982-0d0323b30552-crio-socket\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.891332 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:53.891225 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm podName:dbe0c436-b6d1-4773-9982-0d0323b30552 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:54.391185549 +0000 UTC m=+243.387488763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5wv7t" (UID: "dbe0c436-b6d1-4773-9982-0d0323b30552") : configmap "kube-rbac-proxy" not found Jul 02 20:18:53.891332 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:53.891232 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:18:53.891332 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:53.891292 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls podName:dbe0c436-b6d1-4773-9982-0d0323b30552 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:54.391280693 +0000 UTC m=+243.387583904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wv7t" (UID: "dbe0c436-b6d1-4773-9982-0d0323b30552") : secret "insights-runtime-extractor-tls" not found Jul 02 20:18:53.891516 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.891474 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbe0c436-b6d1-4773-9982-0d0323b30552-data-volume\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.900866 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.900843 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2m55\" (UniqueName: \"kubernetes.io/projected/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-api-access-c2m55\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:53.974569 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:53.974494 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:54.077414 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:54.077381 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:54.186259 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:54.186203 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:54.295532 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:54.295431 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:54.382798 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:54.382767 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/1.log" Jul 02 20:18:54.383151 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:54.383063 2565 scope.go:117] "RemoveContainer" containerID="88be8e09b0c86c045f168630e6235668cd4ccba1ae8dad715c0eaeb35b62c708" Jul 02 20:18:54.383248 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:54.383230 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-7b8w7_openshift-console-operator(5ba6abb2-3ddd-4296-9858-6de372075f3a)\"" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" podUID="5ba6abb2-3ddd-4296-9858-6de372075f3a" Jul 02 20:18:54.394956 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:54.394930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:54.395091 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:54.395023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:54.395091 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:54.395028 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:18:54.395091 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:54.395087 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:18:54.395271 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:54.395106 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls podName:dbe0c436-b6d1-4773-9982-0d0323b30552 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:55.395090076 +0000 UTC m=+244.391393301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wv7t" (UID: "dbe0c436-b6d1-4773-9982-0d0323b30552") : secret "insights-runtime-extractor-tls" not found Jul 02 20:18:54.395271 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:54.395134 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm podName:dbe0c436-b6d1-4773-9982-0d0323b30552 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:55.39512017 +0000 UTC m=+244.391423381 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5wv7t" (UID: "dbe0c436-b6d1-4773-9982-0d0323b30552") : configmap "kube-rbac-proxy" not found Jul 02 20:18:54.395517 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:54.395485 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:54.627982 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:54.627903 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:54.731355 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:54.731325 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:55.405268 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:55.405233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:55.405693 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:55.405339 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:18:55.405693 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:55.405338 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:55.405693 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:55.405405 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm podName:dbe0c436-b6d1-4773-9982-0d0323b30552 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:57.405390833 +0000 UTC m=+246.401694044 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5wv7t" (UID: "dbe0c436-b6d1-4773-9982-0d0323b30552") : configmap "kube-rbac-proxy" not found Jul 02 20:18:55.405693 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:55.405409 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:18:55.405693 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:55.405504 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls podName:dbe0c436-b6d1-4773-9982-0d0323b30552 nodeName:}" failed. No retries permitted until 2026-07-02 20:18:57.405489021 +0000 UTC m=+246.401792231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wv7t" (UID: "dbe0c436-b6d1-4773-9982-0d0323b30552") : secret "insights-runtime-extractor-tls" not found Jul 02 20:18:55.758877 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:55.758781 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8thwh_d462c3b4-ffb3-4449-aff4-cd348fbc835c/dns-node-resolver/0.log" Jul 02 20:18:56.226467 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.226440 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-fnhjv"] Jul 02 20:18:56.229527 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.229512 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-fnhjv" Jul 02 20:18:56.232306 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.232280 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 02 20:18:56.232500 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.232485 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pqddj\"" Jul 02 20:18:56.233514 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.233497 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 02 20:18:56.233596 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.233529 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 02 20:18:56.233596 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.233538 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 02 20:18:56.240379 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.240360 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-fnhjv"] Jul 02 20:18:56.303776 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.303747 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:56.312850 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.312828 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/19efcc8e-aa14-498e-b65a-e47cef8721b0-signing-key\") pod \"service-ca-758f599c5-fnhjv\" (UID: \"19efcc8e-aa14-498e-b65a-e47cef8721b0\") " pod="openshift-service-ca/service-ca-758f599c5-fnhjv" Jul 02 20:18:56.312910 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.312853 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/19efcc8e-aa14-498e-b65a-e47cef8721b0-signing-cabundle\") pod \"service-ca-758f599c5-fnhjv\" (UID: \"19efcc8e-aa14-498e-b65a-e47cef8721b0\") " pod="openshift-service-ca/service-ca-758f599c5-fnhjv" Jul 02 20:18:56.312983 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.312968 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4d7s\" (UniqueName: \"kubernetes.io/projected/19efcc8e-aa14-498e-b65a-e47cef8721b0-kube-api-access-v4d7s\") pod \"service-ca-758f599c5-fnhjv\" (UID: \"19efcc8e-aa14-498e-b65a-e47cef8721b0\") " pod="openshift-service-ca/service-ca-758f599c5-fnhjv" Jul 02 20:18:56.413740 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.413700 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/19efcc8e-aa14-498e-b65a-e47cef8721b0-signing-key\") pod \"service-ca-758f599c5-fnhjv\" (UID: \"19efcc8e-aa14-498e-b65a-e47cef8721b0\") " pod="openshift-service-ca/service-ca-758f599c5-fnhjv" Jul 02 20:18:56.413740 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.413739 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/19efcc8e-aa14-498e-b65a-e47cef8721b0-signing-cabundle\") pod \"service-ca-758f599c5-fnhjv\" (UID: \"19efcc8e-aa14-498e-b65a-e47cef8721b0\") " pod="openshift-service-ca/service-ca-758f599c5-fnhjv" Jul 02 20:18:56.414120 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.413843 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4d7s\" (UniqueName: \"kubernetes.io/projected/19efcc8e-aa14-498e-b65a-e47cef8721b0-kube-api-access-v4d7s\") pod \"service-ca-758f599c5-fnhjv\" (UID: \"19efcc8e-aa14-498e-b65a-e47cef8721b0\") " pod="openshift-service-ca/service-ca-758f599c5-fnhjv" Jul 02 20:18:56.414491 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.414474 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/19efcc8e-aa14-498e-b65a-e47cef8721b0-signing-cabundle\") pod \"service-ca-758f599c5-fnhjv\" (UID: \"19efcc8e-aa14-498e-b65a-e47cef8721b0\") " pod="openshift-service-ca/service-ca-758f599c5-fnhjv" Jul 02 20:18:56.416184 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.416158 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/19efcc8e-aa14-498e-b65a-e47cef8721b0-signing-key\") pod \"service-ca-758f599c5-fnhjv\" (UID: \"19efcc8e-aa14-498e-b65a-e47cef8721b0\") " pod="openshift-service-ca/service-ca-758f599c5-fnhjv" Jul 02 20:18:56.422045 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.422027 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4d7s\" (UniqueName: \"kubernetes.io/projected/19efcc8e-aa14-498e-b65a-e47cef8721b0-kube-api-access-v4d7s\") pod \"service-ca-758f599c5-fnhjv\" (UID: \"19efcc8e-aa14-498e-b65a-e47cef8721b0\") " pod="openshift-service-ca/service-ca-758f599c5-fnhjv" Jul 02 20:18:56.447530 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.447507 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:56.514459 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.514386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:18:56.514459 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.514427 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:18:56.514459 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.514455 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tdq67\" (UID: \"cabf7e15-b90b-4363-939a-ac9f61cc07ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:18:56.514689 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:56.514537 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:18:56.514689 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:56.514542 2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 20:18:56.514689 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:56.514547 2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 20:18:56.514689 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:56.514560 2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7b5cf7cccf-9vswd: secret "image-registry-tls" not found Jul 02 20:18:56.514689 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:56.514589 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls podName:a7e7fea8-aaec-47ae-9137-5be0b93671bb nodeName:}" failed. No retries permitted until 2026-07-02 20:19:04.514575421 +0000 UTC m=+253.510878632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mqr6k" (UID: "a7e7fea8-aaec-47ae-9137-5be0b93671bb") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:18:56.514689 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:56.514610 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls podName:e20f83f3-3574-41f1-99c9-0e22df9a2e0d nodeName:}" failed. No retries permitted until 2026-07-02 20:19:04.514594391 +0000 UTC m=+253.510897617 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls") pod "image-registry-7b5cf7cccf-9vswd" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d") : secret "image-registry-tls" not found Jul 02 20:18:56.514689 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:56.514628 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls podName:cabf7e15-b90b-4363-939a-ac9f61cc07ac nodeName:}" failed. No retries permitted until 2026-07-02 20:19:04.51461804 +0000 UTC m=+253.510921251 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-tdq67" (UID: "cabf7e15-b90b-4363-939a-ac9f61cc07ac") : secret "samples-operator-tls" not found Jul 02 20:18:56.538462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.538442 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-fnhjv" Jul 02 20:18:56.660070 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.660034 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-fnhjv"] Jul 02 20:18:56.662775 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:18:56.662748 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19efcc8e_aa14_498e_b65a_e47cef8721b0.slice/crio-b61428f104bace8d093eaeac9db803aaa344fc7a1665faca64b18c5352878d1b WatchSource:0}: Error finding container b61428f104bace8d093eaeac9db803aaa344fc7a1665faca64b18c5352878d1b: Status 404 returned error can't find the container with id b61428f104bace8d093eaeac9db803aaa344fc7a1665faca64b18c5352878d1b Jul 02 20:18:56.759371 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:56.759347 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jz4g9_a9ef4988-0ad9-491c-9935-929c5a0eb8ba/node-ca/0.log" Jul 02 20:18:57.186070 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:57.186029 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:18:57.362033 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:57.362002 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gj22w_efc20391-221b-452a-8960-bd8d55818dc9/serve-healthcheck-canary/0.log" Jul 02 20:18:57.393032 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:57.393000 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-fnhjv" event={"ID":"19efcc8e-aa14-498e-b65a-e47cef8721b0","Type":"ContainerStarted","Data":"b61428f104bace8d093eaeac9db803aaa344fc7a1665faca64b18c5352878d1b"} Jul 02 20:18:57.420508 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:57.420472 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:57.420947 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:57.420556 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:18:57.420947 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:57.420644 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:18:57.420947 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:57.420694 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:18:57.420947 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:57.420732 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls podName:dbe0c436-b6d1-4773-9982-0d0323b30552 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:01.420711345 +0000 UTC m=+250.417014557 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wv7t" (UID: "dbe0c436-b6d1-4773-9982-0d0323b30552") : secret "insights-runtime-extractor-tls" not found Jul 02 20:18:57.420947 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:57.420748 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm podName:dbe0c436-b6d1-4773-9982-0d0323b30552 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:01.420741008 +0000 UTC m=+250.417044219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5wv7t" (UID: "dbe0c436-b6d1-4773-9982-0d0323b30552") : configmap "kube-rbac-proxy" not found Jul 02 20:18:59.085850 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:59.085753 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:59.086188 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:59.085856 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:18:59.086291 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:59.086277 2565 scope.go:117] "RemoveContainer" containerID="88be8e09b0c86c045f168630e6235668cd4ccba1ae8dad715c0eaeb35b62c708" Jul 02 20:18:59.086473 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:18:59.086451 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-7b8w7_openshift-console-operator(5ba6abb2-3ddd-4296-9858-6de372075f3a)\"" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" podUID="5ba6abb2-3ddd-4296-9858-6de372075f3a" Jul 02 20:18:59.401551 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:59.401519 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-fnhjv" event={"ID":"19efcc8e-aa14-498e-b65a-e47cef8721b0","Type":"ContainerStarted","Data":"eb43be790c539708b4903f9ae8758525fdcccfc38195dffe6a73c3c5da2658f8"} Jul 02 20:18:59.418204 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:59.418159 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-fnhjv" podStartSLOduration=1.29459045 podStartE2EDuration="3.418145747s" totalImagesPullingTime="2.123555297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:18:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:18:59.417271532 +0000 UTC m=+248.413574871" watchObservedRunningTime="2026-07-02 20:18:59.418145747 +0000 UTC m=+248.414448980" Jul 02 20:18:59.831077 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:18:59.830998 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:00.504584 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:00.504547 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:00.605059 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:00.605022 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:00.730814 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:00.730783 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:00.834913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:00.834822 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:00.936764 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:00.936734 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:01.061977 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:01.061944 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:01.408507 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:01.408470 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:01.453703 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:01.453670 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:19:01.453877 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:01.453768 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:19:01.453877 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:01.453823 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:19:01.454023 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:01.453879 2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 20:19:01.454023 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:01.453886 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls podName:dbe0c436-b6d1-4773-9982-0d0323b30552 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:09.453871119 +0000 UTC m=+258.450174329 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wv7t" (UID: "dbe0c436-b6d1-4773-9982-0d0323b30552") : secret "insights-runtime-extractor-tls" not found Jul 02 20:19:01.454023 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:01.453928 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm podName:dbe0c436-b6d1-4773-9982-0d0323b30552 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:09.453912284 +0000 UTC m=+258.450215501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5wv7t" (UID: "dbe0c436-b6d1-4773-9982-0d0323b30552") : configmap "kube-rbac-proxy" not found Jul 02 20:19:02.203165 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:02.203133 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:02.601275 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:02.601182 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:02.822184 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:02.822150 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:03.002016 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:03.001989 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:03.206334 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:03.206301 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:03.265531 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:03.265444 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:19:03.265687 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:03.265641 2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 02 20:19:03.265763 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:03.265750 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs podName:8caf8e2d-384f-4ecb-a60c-3ce8137a419a nodeName:}" failed. No retries permitted until 2026-07-02 20:21:05.265729119 +0000 UTC m=+374.262032336 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs") pod "network-metrics-daemon-hr888" (UID: "8caf8e2d-384f-4ecb-a60c-3ce8137a419a") : secret "metrics-daemon-secret" not found Jul 02 20:19:03.398533 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:03.398499 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:03.815320 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:03.815281 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:04.198661 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.198631 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:04.397756 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.397721 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:04.450736 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.450659 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:04.575878 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.575851 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:19:04.576061 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.575891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:19:04.576061 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.575913 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tdq67\" (UID: \"cabf7e15-b90b-4363-939a-ac9f61cc07ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:19:04.576061 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:04.576015 2565 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 02 20:19:04.576232 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:04.576083 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls podName:a7e7fea8-aaec-47ae-9137-5be0b93671bb nodeName:}" failed. No retries permitted until 2026-07-02 20:19:20.576063725 +0000 UTC m=+269.572366939 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-mqr6k" (UID: "a7e7fea8-aaec-47ae-9137-5be0b93671bb") : secret "cluster-monitoring-operator-tls" not found Jul 02 20:19:04.578406 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.578387 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls\") pod \"image-registry-7b5cf7cccf-9vswd\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:19:04.578476 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.578387 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cabf7e15-b90b-4363-939a-ac9f61cc07ac-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-tdq67\" (UID: \"cabf7e15-b90b-4363-939a-ac9f61cc07ac\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:19:04.666913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.666885 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q9zqq\"" Jul 02 20:19:04.675104 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.675085 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" Jul 02 20:19:04.681538 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.681519 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s7jgs\"" Jul 02 20:19:04.689814 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.689790 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:19:04.806856 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.806826 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:04.823890 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.823703 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67"] Jul 02 20:19:04.841392 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:04.841366 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b5cf7cccf-9vswd"] Jul 02 20:19:04.843125 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:19:04.843099 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode20f83f3_3574_41f1_99c9_0e22df9a2e0d.slice/crio-36f1329fa7a5c2865c6379e2fe0659e5b989fe80b569e7e660bd5c493f32c091 WatchSource:0}: Error finding container 36f1329fa7a5c2865c6379e2fe0659e5b989fe80b569e7e660bd5c493f32c091: Status 404 returned error can't find the container with id 36f1329fa7a5c2865c6379e2fe0659e5b989fe80b569e7e660bd5c493f32c091 Jul 02 20:19:05.200022 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:05.199989 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:05.401299 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:05.401265 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:05.421489 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:05.421454 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" event={"ID":"e20f83f3-3574-41f1-99c9-0e22df9a2e0d","Type":"ContainerStarted","Data":"decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89"} Jul 02 20:19:05.421615 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:05.421504 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" event={"ID":"e20f83f3-3574-41f1-99c9-0e22df9a2e0d","Type":"ContainerStarted","Data":"36f1329fa7a5c2865c6379e2fe0659e5b989fe80b569e7e660bd5c493f32c091"} Jul 02 20:19:05.421615 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:05.421573 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:19:05.422612 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:05.422578 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" event={"ID":"cabf7e15-b90b-4363-939a-ac9f61cc07ac","Type":"ContainerStarted","Data":"c300d0a1de4d6f013ac71544b08a5b412ecfd71696780f996e24df64bcfd3924"} Jul 02 20:19:05.443910 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:05.443868 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" podStartSLOduration=17.443855628 podStartE2EDuration="17.443855628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:05.442423571 +0000 UTC m=+254.438726804" watchObservedRunningTime="2026-07-02 20:19:05.443855628 +0000 UTC m=+254.440158838" Jul 02 20:19:05.801299 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:05.801272 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:06.429255 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:06.429203 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:06.799498 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:06.799416 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:06.902898 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:06.902862 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:06.964854 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:06.964818 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:07.202447 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:07.202420 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:07.430487 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:07.430452 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" event={"ID":"cabf7e15-b90b-4363-939a-ac9f61cc07ac","Type":"ContainerStarted","Data":"e117adc34e429c75950e3ff15b0354d7b259ec5800294ffacb47d7a9bc648075"} Jul 02 20:19:07.430833 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:07.430510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" event={"ID":"cabf7e15-b90b-4363-939a-ac9f61cc07ac","Type":"ContainerStarted","Data":"840744be6077ba8de88b8bcdd38d962fe9e2b293fe55b46665d62d037fdfc543"} Jul 02 20:19:07.598531 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:07.598425 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:07.809528 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:07.809491 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:08.202564 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:08.202535 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:08.811109 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:08.811074 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:09.197836 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:09.197802 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:09.398272 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:09.398244 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:09.499340 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:09.499256 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:09.518492 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:09.518463 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:19:09.518564 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:09.518539 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:19:09.518612 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:09.518600 2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 20:19:09.518682 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:09.518672 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls podName:dbe0c436-b6d1-4773-9982-0d0323b30552 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:25.518655817 +0000 UTC m=+274.514959029 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5wv7t" (UID: "dbe0c436-b6d1-4773-9982-0d0323b30552") : secret "insights-runtime-extractor-tls" not found Jul 02 20:19:09.519121 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:09.519104 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dbe0c436-b6d1-4773-9982-0d0323b30552-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:19:09.600512 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:09.600483 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:09.998467 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:09.998432 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:10.601715 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:10.601683 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:10.807430 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:10.807396 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:11.001590 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:11.001555 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:11.297572 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:11.297475 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:13.565875 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:13.565846 2565 scope.go:117] "RemoveContainer" containerID="88be8e09b0c86c045f168630e6235668cd4ccba1ae8dad715c0eaeb35b62c708" Jul 02 20:19:14.453734 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:14.453708 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:19:14.454099 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:14.454085 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/1.log" Jul 02 20:19:14.454147 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:14.454117 2565 generic.go:356] "Generic (PLEG): container finished" podID="5ba6abb2-3ddd-4296-9858-6de372075f3a" containerID="294184b6ff5c5657a5552a3ccd5d6a5bcb63d679cdc5aea4a5fd70838a2a912a" exitCode=255 Jul 02 20:19:14.454180 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:14.454151 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" event={"ID":"5ba6abb2-3ddd-4296-9858-6de372075f3a","Type":"ContainerDied","Data":"294184b6ff5c5657a5552a3ccd5d6a5bcb63d679cdc5aea4a5fd70838a2a912a"} Jul 02 20:19:14.454228 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:14.454191 2565 scope.go:117] "RemoveContainer" containerID="88be8e09b0c86c045f168630e6235668cd4ccba1ae8dad715c0eaeb35b62c708" Jul 02 20:19:14.454564 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:14.454537 2565 scope.go:117] "RemoveContainer" containerID="294184b6ff5c5657a5552a3ccd5d6a5bcb63d679cdc5aea4a5fd70838a2a912a" Jul 02 20:19:14.454730 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:14.454711 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-7b8w7_openshift-console-operator(5ba6abb2-3ddd-4296-9858-6de372075f3a)\"" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" podUID="5ba6abb2-3ddd-4296-9858-6de372075f3a" Jul 02 20:19:14.471956 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:14.471915 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-tdq67" podStartSLOduration=24.133951765 podStartE2EDuration="26.471901176s" totalImagesPullingTime="2.337949411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:07.451059611 +0000 UTC m=+256.447362843" watchObservedRunningTime="2026-07-02 20:19:14.471901176 +0000 UTC m=+263.468204409" Jul 02 20:19:15.458404 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:15.458378 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:19:17.020396 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:17.020360 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b5cf7cccf-9vswd"] Jul 02 20:19:17.111538 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:17.111494 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:17.209549 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:17.209520 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:19.085488 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:19.085441 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:19:19.085889 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:19.085519 2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:19:19.085889 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:19.085864 2565 scope.go:117] "RemoveContainer" containerID="294184b6ff5c5657a5552a3ccd5d6a5bcb63d679cdc5aea4a5fd70838a2a912a" Jul 02 20:19:19.086045 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:19.086024 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-7b8w7_openshift-console-operator(5ba6abb2-3ddd-4296-9858-6de372075f3a)\"" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" podUID="5ba6abb2-3ddd-4296-9858-6de372075f3a" Jul 02 20:19:19.328904 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:19.328875 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:19.383073 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:19.382989 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:20.604866 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:20.604828 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:19:20.607370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:20.607348 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7e7fea8-aaec-47ae-9137-5be0b93671bb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-mqr6k\" (UID: \"a7e7fea8-aaec-47ae-9137-5be0b93671bb\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:19:20.873436 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:20.873360 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-j5mmh\"" Jul 02 20:19:20.880630 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:20.880613 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" Jul 02 20:19:21.019428 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:21.017935 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k"] Jul 02 20:19:21.023355 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:19:21.023315 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e7fea8_aaec_47ae_9137_5be0b93671bb.slice/crio-e06a60205c3646801f1ecfbc082132b10b784bce1b3056eafcc82af9423eb54e WatchSource:0}: Error finding container e06a60205c3646801f1ecfbc082132b10b784bce1b3056eafcc82af9423eb54e: Status 404 returned error can't find the container with id e06a60205c3646801f1ecfbc082132b10b784bce1b3056eafcc82af9423eb54e Jul 02 20:19:21.043792 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:21.043759 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:21.107752 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:21.107718 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:21.477571 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:21.477531 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" event={"ID":"a7e7fea8-aaec-47ae-9137-5be0b93671bb","Type":"ContainerStarted","Data":"e06a60205c3646801f1ecfbc082132b10b784bce1b3056eafcc82af9423eb54e"} Jul 02 20:19:23.486006 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:23.485974 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" event={"ID":"a7e7fea8-aaec-47ae-9137-5be0b93671bb","Type":"ContainerStarted","Data":"930e62bd8c42771681f5dcd8d315d34f86daed62084ef009c73a0cf7b5bfe9b9"} Jul 02 20:19:23.502346 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:23.502295 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-mqr6k" podStartSLOduration=33.243954405 podStartE2EDuration="35.502279229s" totalImagesPullingTime="2.258324824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:23.500740629 +0000 UTC m=+272.497043862" watchObservedRunningTime="2026-07-02 20:19:23.502279229 +0000 UTC m=+272.498582484" Jul 02 20:19:23.589117 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:23.589020 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:23.686433 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:23.686382 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:23.754860 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:23.754835 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:24.132193 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:24.132129 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:25.547980 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:25.547938 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:19:25.550401 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:25.550380 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dbe0c436-b6d1-4773-9982-0d0323b30552-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5wv7t\" (UID: \"dbe0c436-b6d1-4773-9982-0d0323b30552\") " pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:19:25.804752 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:25.804657 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5wv7t" Jul 02 20:19:25.970510 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:25.970479 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5wv7t"] Jul 02 20:19:25.973476 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:19:25.973449 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbe0c436_b6d1_4773_9982_0d0323b30552.slice/crio-6021a0632bf0e858a669ea68e588074784f62c11fece66a5838ca43695fa5a48 WatchSource:0}: Error finding container 6021a0632bf0e858a669ea68e588074784f62c11fece66a5838ca43695fa5a48: Status 404 returned error can't find the container with id 6021a0632bf0e858a669ea68e588074784f62c11fece66a5838ca43695fa5a48 Jul 02 20:19:26.497325 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:26.497288 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5wv7t" event={"ID":"dbe0c436-b6d1-4773-9982-0d0323b30552","Type":"ContainerStarted","Data":"6021a0632bf0e858a669ea68e588074784f62c11fece66a5838ca43695fa5a48"} Jul 02 20:19:27.027520 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.027454 2565 patch_prober.go:28] interesting pod/image-registry-7b5cf7cccf-9vswd container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 02 20:19:27.027983 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.027524 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" podUID="e20f83f3-3574-41f1-99c9-0e22df9a2e0d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 20:19:27.788131 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.788100 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:27.821336 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.821306 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2"] Jul 02 20:19:27.828414 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.828392 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:27.831398 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.831372 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 02 20:19:27.831398 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.831389 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 02 20:19:27.831573 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.831458 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 20:19:27.832552 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.832535 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jr2mv\"" Jul 02 20:19:27.835242 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.835164 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2"] Jul 02 20:19:27.849136 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.849109 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:27.866560 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.866532 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57e179a8-99ec-42a4-875e-784a933b3c82-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:27.866701 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.866605 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:27.866701 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.866640 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:27.866802 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.866732 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9drd5\" (UniqueName: \"kubernetes.io/projected/57e179a8-99ec-42a4-875e-784a933b3c82-kube-api-access-9drd5\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:27.967987 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.967947 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9drd5\" (UniqueName: \"kubernetes.io/projected/57e179a8-99ec-42a4-875e-784a933b3c82-kube-api-access-9drd5\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:27.968159 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.968047 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57e179a8-99ec-42a4-875e-784a933b3c82-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:27.968159 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.968113 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:27.968159 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.968146 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:27.968356 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:27.968293 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 02 20:19:27.968427 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:27.968402 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-tls podName:57e179a8-99ec-42a4-875e-784a933b3c82 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:28.468371216 +0000 UTC m=+277.464674446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-vh5g2" (UID: "57e179a8-99ec-42a4-875e-784a933b3c82") : secret "prometheus-operator-tls" not found Jul 02 20:19:27.968971 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.968880 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/57e179a8-99ec-42a4-875e-784a933b3c82-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:27.970753 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.970729 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:27.977141 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:27.977121 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9drd5\" (UniqueName: \"kubernetes.io/projected/57e179a8-99ec-42a4-875e-784a933b3c82-kube-api-access-9drd5\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:28.471338 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:28.471301 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:28.471795 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:28.471479 2565 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 02 20:19:28.471795 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:28.471561 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-tls podName:57e179a8-99ec-42a4-875e-784a933b3c82 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:29.471539117 +0000 UTC m=+278.467842332 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-vh5g2" (UID: "57e179a8-99ec-42a4-875e-784a933b3c82") : secret "prometheus-operator-tls" not found Jul 02 20:19:28.664615 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:28.664568 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:29.482139 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:29.482108 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:29.484806 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:29.484784 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/57e179a8-99ec-42a4-875e-784a933b3c82-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-vh5g2\" (UID: \"57e179a8-99ec-42a4-875e-784a933b3c82\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:29.640001 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:29.639966 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" Jul 02 20:19:29.853317 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:29.853296 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2"] Jul 02 20:19:29.856661 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:19:29.856624 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e179a8_99ec_42a4_875e_784a933b3c82.slice/crio-fc4996fdd04c16ec55fbdc9aa5a137baecd52d9de9f7a564cc3dfa6913b92cde WatchSource:0}: Error finding container fc4996fdd04c16ec55fbdc9aa5a137baecd52d9de9f7a564cc3dfa6913b92cde: Status 404 returned error can't find the container with id fc4996fdd04c16ec55fbdc9aa5a137baecd52d9de9f7a564cc3dfa6913b92cde Jul 02 20:19:30.512433 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:30.512402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" event={"ID":"57e179a8-99ec-42a4-875e-784a933b3c82","Type":"ContainerStarted","Data":"fc4996fdd04c16ec55fbdc9aa5a137baecd52d9de9f7a564cc3dfa6913b92cde"} Jul 02 20:19:30.514076 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:30.514043 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5wv7t" event={"ID":"dbe0c436-b6d1-4773-9982-0d0323b30552","Type":"ContainerStarted","Data":"7d897b2166ff3cc3e7f46ebd2b1d40719cc1c9fb6ca462751ebcb8fad0872d58"} Jul 02 20:19:30.514186 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:30.514096 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5wv7t" event={"ID":"dbe0c436-b6d1-4773-9982-0d0323b30552","Type":"ContainerStarted","Data":"74e81bfcff19fa5796f26e5c4cea290ee3b634c79b0b6bdeab7f4c336a0ac5e2"} Jul 02 20:19:31.520559 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:31.520491 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" event={"ID":"57e179a8-99ec-42a4-875e-784a933b3c82","Type":"ContainerStarted","Data":"f7fcb6190ac94771215c594a85ff302ea34eb2eb0da6766a4716cccfbacce0f8"} Jul 02 20:19:31.520559 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:31.520544 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" event={"ID":"57e179a8-99ec-42a4-875e-784a933b3c82","Type":"ContainerStarted","Data":"bfa6abaa2ccc393f3ae8efc6b579a83e5b5030035d26082eb3c5fc368b2aa341"} Jul 02 20:19:31.535837 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:31.535805 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:31.539832 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:31.539790 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-vh5g2" podStartSLOduration=3.155296308 podStartE2EDuration="4.539775052s" totalImagesPullingTime="1.384478744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:19:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:31.538442243 +0000 UTC m=+280.534745478" watchObservedRunningTime="2026-07-02 20:19:31.539775052 +0000 UTC m=+280.536078284" Jul 02 20:19:31.567855 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:31.567833 2565 scope.go:117] "RemoveContainer" containerID="294184b6ff5c5657a5552a3ccd5d6a5bcb63d679cdc5aea4a5fd70838a2a912a" Jul 02 20:19:31.568043 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:31.568020 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-7b8w7_openshift-console-operator(5ba6abb2-3ddd-4296-9858-6de372075f3a)\"" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" podUID="5ba6abb2-3ddd-4296-9858-6de372075f3a" Jul 02 20:19:31.636772 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:31.636737 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:31.736468 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:31.736428 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:32.917830 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:32.917797 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:33.035527 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.035484 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:33.141573 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.141482 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:33.152050 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.152026 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9"] Jul 02 20:19:33.156365 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.156335 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq"] Jul 02 20:19:33.156486 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.156466 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.158844 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.158824 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.159356 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.159006 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 02 20:19:33.159356 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.159291 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 02 20:19:33.159356 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.159304 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-4vvjm\"" Jul 02 20:19:33.162443 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.162422 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 02 20:19:33.162443 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.162434 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 02 20:19:33.162669 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.162642 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-46vb2\"" Jul 02 20:19:33.162669 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.162666 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 02 20:19:33.168059 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.166486 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r9qvf"] Jul 02 20:19:33.169123 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.169102 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9"] Jul 02 20:19:33.169275 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.169259 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.169647 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.169620 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq"] Jul 02 20:19:33.172341 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.171973 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 20:19:33.172341 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.171998 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 20:19:33.172341 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.172120 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 20:19:33.172341 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.172248 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m47xv\"" Jul 02 20:19:33.209895 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.209874 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:33.216898 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.216877 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb8v6\" (UniqueName: \"kubernetes.io/projected/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-api-access-nb8v6\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.216999 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.216907 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.216999 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.216932 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.216999 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.216959 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.217136 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217014 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/711c2aad-274f-4c23-8280-9c9aebefec56-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.217136 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217060 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.217136 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217103 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.217283 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217151 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.217283 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217180 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.217283 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217229 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvhn\" (UniqueName: \"kubernetes.io/projected/711c2aad-274f-4c23-8280-9c9aebefec56-kube-api-access-8zvhn\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.217392 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217284 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5256x\" (UniqueName: \"kubernetes.io/projected/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-kube-api-access-5256x\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.217392 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217317 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-textfile\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.217392 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217345 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-sys\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.217537 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217421 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.217537 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217456 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-root\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.217537 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217507 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.217698 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217542 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-tls\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.217698 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217582 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-wtmp\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.217698 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.217618 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-metrics-client-ca\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.318423 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318386 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.318423 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318424 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.318679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318458 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.318679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318483 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.318679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318510 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvhn\" (UniqueName: \"kubernetes.io/projected/711c2aad-274f-4c23-8280-9c9aebefec56-kube-api-access-8zvhn\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.318679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318536 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5256x\" (UniqueName: \"kubernetes.io/projected/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-kube-api-access-5256x\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.318679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318572 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-textfile\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.318679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318596 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-sys\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.318679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318641 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.318679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318659 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-sys\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.318679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318668 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-root\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.319106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318709 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-root\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.319106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318722 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.319106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318761 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-tls\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.319106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318792 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-wtmp\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.319106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318814 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-metrics-client-ca\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.319106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318848 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb8v6\" (UniqueName: \"kubernetes.io/projected/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-api-access-nb8v6\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.319106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318872 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.319106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318899 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.319106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318925 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.319106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.318954 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/711c2aad-274f-4c23-8280-9c9aebefec56-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.319106 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.319049 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-wtmp\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.319619 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.319365 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-textfile\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.319619 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.319421 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.319619 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:33.319453 2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 02 20:19:33.319619 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:33.319506 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-tls podName:b8b6f017-988d-4d7f-930a-aaf71f52a7f2 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:33.819487053 +0000 UTC m=+282.815790270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-tls") pod "node-exporter-r9qvf" (UID: "b8b6f017-988d-4d7f-930a-aaf71f52a7f2") : secret "node-exporter-tls" not found Jul 02 20:19:33.319619 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:33.319512 2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 02 20:19:33.319619 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:33.319567 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-tls podName:0cff93e5-3bd8-4d6d-a05d-3649be4e11a5 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:33.819550644 +0000 UTC m=+282.815853861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-q24zq" (UID: "0cff93e5-3bd8-4d6d-a05d-3649be4e11a5") : secret "kube-state-metrics-tls" not found Jul 02 20:19:33.319922 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:33.319709 2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 02 20:19:33.319922 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.319714 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/711c2aad-274f-4c23-8280-9c9aebefec56-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.319922 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.319724 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.319922 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:33.319746 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-tls podName:711c2aad-274f-4c23-8280-9c9aebefec56 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:33.819733992 +0000 UTC m=+282.816037210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-shzp9" (UID: "711c2aad-274f-4c23-8280-9c9aebefec56") : secret "openshift-state-metrics-tls" not found Jul 02 20:19:33.320115 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.320014 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-metrics-client-ca\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.320335 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.320290 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-accelerators-collector-config\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.320458 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.320429 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.322474 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.322450 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.322603 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.322556 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.322686 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.322666 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.327912 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.327887 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5256x\" (UniqueName: \"kubernetes.io/projected/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-kube-api-access-5256x\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.328431 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.328413 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb8v6\" (UniqueName: \"kubernetes.io/projected/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-api-access-nb8v6\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.328491 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.328442 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvhn\" (UniqueName: \"kubernetes.io/projected/711c2aad-274f-4c23-8280-9c9aebefec56-kube-api-access-8zvhn\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.413474 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.413445 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:33.524282 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.524252 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:33.529825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.529796 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5wv7t" event={"ID":"dbe0c436-b6d1-4773-9982-0d0323b30552","Type":"ContainerStarted","Data":"dbdc1752a8bc918ec4f3e7040fbbacd3d72c0d6ca532ea15d11bef9aa07a83c5"} Jul 02 20:19:33.548704 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.548661 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5wv7t" podStartSLOduration=33.766950859 podStartE2EDuration="40.548645005s" totalImagesPullingTime="6.781694146s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:18:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:33.548015185 +0000 UTC m=+282.544318418" watchObservedRunningTime="2026-07-02 20:19:33.548645005 +0000 UTC m=+282.544948234" Jul 02 20:19:33.823357 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.823259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:33.823357 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.823313 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:33.823580 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:33.823409 2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 02 20:19:33.823580 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.823430 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-tls\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.823580 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:33.823456 2565 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 02 20:19:33.823580 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:33.823480 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-tls podName:0cff93e5-3bd8-4d6d-a05d-3649be4e11a5 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:34.82346048 +0000 UTC m=+283.819763715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-q24zq" (UID: "0cff93e5-3bd8-4d6d-a05d-3649be4e11a5") : secret "kube-state-metrics-tls" not found Jul 02 20:19:33.823580 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:33.823506 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-tls podName:711c2aad-274f-4c23-8280-9c9aebefec56 nodeName:}" failed. No retries permitted until 2026-07-02 20:19:34.823493214 +0000 UTC m=+283.819796425 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-shzp9" (UID: "711c2aad-274f-4c23-8280-9c9aebefec56") : secret "openshift-state-metrics-tls" not found Jul 02 20:19:33.826147 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.826128 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8b6f017-988d-4d7f-930a-aaf71f52a7f2-node-exporter-tls\") pod \"node-exporter-r9qvf\" (UID: \"b8b6f017-988d-4d7f-930a-aaf71f52a7f2\") " pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:33.890545 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.890515 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:33.991478 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:33.991445 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:34.084508 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.084418 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r9qvf" Jul 02 20:19:34.095547 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:19:34.095517 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b6f017_988d_4d7f_930a_aaf71f52a7f2.slice/crio-595419d23d1b19b7bd4bc16c58ed168b4b6973b4f089c5eb8feacad51678f1c2 WatchSource:0}: Error finding container 595419d23d1b19b7bd4bc16c58ed168b4b6973b4f089c5eb8feacad51678f1c2: Status 404 returned error can't find the container with id 595419d23d1b19b7bd4bc16c58ed168b4b6973b4f089c5eb8feacad51678f1c2 Jul 02 20:19:34.144652 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.144620 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:34.172373 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:34.167279 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-hmr7z" podUID="3386b537-14c7-4ed3-9c7c-7d5d83e7ae93" Jul 02 20:19:34.234385 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.234353 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:34.358882 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.358811 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:34.456801 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.456768 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:34.534233 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.534175 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9qvf" event={"ID":"b8b6f017-988d-4d7f-930a-aaf71f52a7f2","Type":"ContainerStarted","Data":"595419d23d1b19b7bd4bc16c58ed168b4b6973b4f089c5eb8feacad51678f1c2"} Jul 02 20:19:34.534429 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.534189 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmr7z" Jul 02 20:19:34.834696 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.834662 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:34.834696 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.834701 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:34.837184 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.837152 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0cff93e5-3bd8-4d6d-a05d-3649be4e11a5-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-q24zq\" (UID: \"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:34.837327 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.837306 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/711c2aad-274f-4c23-8280-9c9aebefec56-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-shzp9\" (UID: \"711c2aad-274f-4c23-8280-9c9aebefec56\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:34.970928 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.970894 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" Jul 02 20:19:34.978204 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:34.978176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" Jul 02 20:19:35.131452 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:35.131374 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9"] Jul 02 20:19:35.153023 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:35.152944 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq"] Jul 02 20:19:35.317279 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:19:35.317248 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711c2aad_274f_4c23_8280_9c9aebefec56.slice/crio-e66b74265287f8e644a04a511a2f4af037ae3a4b59fb445eeb266119af5f2479 WatchSource:0}: Error finding container e66b74265287f8e644a04a511a2f4af037ae3a4b59fb445eeb266119af5f2479: Status 404 returned error can't find the container with id e66b74265287f8e644a04a511a2f4af037ae3a4b59fb445eeb266119af5f2479 Jul 02 20:19:35.317731 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:19:35.317702 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cff93e5_3bd8_4d6d_a05d_3649be4e11a5.slice/crio-4c85194572f6f6b6e0561a8d0636309eb74cbff5216e000837bbfacae489f2dd WatchSource:0}: Error finding container 4c85194572f6f6b6e0561a8d0636309eb74cbff5216e000837bbfacae489f2dd: Status 404 returned error can't find the container with id 4c85194572f6f6b6e0561a8d0636309eb74cbff5216e000837bbfacae489f2dd Jul 02 20:19:35.539521 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:35.539480 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" event={"ID":"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5","Type":"ContainerStarted","Data":"4c85194572f6f6b6e0561a8d0636309eb74cbff5216e000837bbfacae489f2dd"} Jul 02 20:19:35.541014 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:35.540986 2565 generic.go:356] "Generic (PLEG): container finished" podID="b8b6f017-988d-4d7f-930a-aaf71f52a7f2" containerID="ced24388e627271c15f107aed47b5ded69a55c94c28034f8b1e33fb23c538e55" exitCode=0 Jul 02 20:19:35.541115 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:35.541069 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9qvf" event={"ID":"b8b6f017-988d-4d7f-930a-aaf71f52a7f2","Type":"ContainerDied","Data":"ced24388e627271c15f107aed47b5ded69a55c94c28034f8b1e33fb23c538e55"} Jul 02 20:19:35.542982 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:35.542962 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" event={"ID":"711c2aad-274f-4c23-8280-9c9aebefec56","Type":"ContainerStarted","Data":"fe4df8afe03fa9028a16e1956eb80ba139c4a7b633c301823dfeba190105f417"} Jul 02 20:19:35.543073 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:35.543003 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" event={"ID":"711c2aad-274f-4c23-8280-9c9aebefec56","Type":"ContainerStarted","Data":"666e723cbb9d159127f49d8da2b783668a95411fe21823655b42ad8d8dbea696"} Jul 02 20:19:35.543073 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:35.543031 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" event={"ID":"711c2aad-274f-4c23-8280-9c9aebefec56","Type":"ContainerStarted","Data":"e66b74265287f8e644a04a511a2f4af037ae3a4b59fb445eeb266119af5f2479"} Jul 02 20:19:36.218234 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:36.218156 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:36.549786 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:36.549751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9qvf" event={"ID":"b8b6f017-988d-4d7f-930a-aaf71f52a7f2","Type":"ContainerStarted","Data":"803a59ef3a82b9efa2ef09e8962fa03d377831b9b0e80491c7f8e482fefb0d2a"} Jul 02 20:19:36.549975 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:36.549814 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r9qvf" event={"ID":"b8b6f017-988d-4d7f-930a-aaf71f52a7f2","Type":"ContainerStarted","Data":"bdf9c61f202ca8b46c7369431ce12fb9de2d0cc6a0fd3128411260c0c92dd633"} Jul 02 20:19:36.570727 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:36.570665 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r9qvf" podStartSLOduration=2.242814534 podStartE2EDuration="3.570647303s" totalImagesPullingTime="1.248545296s" totalInitContainerRuntime="79.287473ms" isStatefulPod=true podCreationTimestamp="2026-07-02 20:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:36.568949593 +0000 UTC m=+285.565252831" watchObservedRunningTime="2026-07-02 20:19:36.570647303 +0000 UTC m=+285.566950536" Jul 02 20:19:36.573607 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:36.573578 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:36.667261 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:36.667229 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:37.024832 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:37.024801 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:19:37.555402 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:37.555357 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" event={"ID":"711c2aad-274f-4c23-8280-9c9aebefec56","Type":"ContainerStarted","Data":"0855031781c9b3c63c9538361c79a74e8d8eb92f2edc986ab05b44d0be5693ac"} Jul 02 20:19:37.557563 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:37.557525 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" event={"ID":"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5","Type":"ContainerStarted","Data":"4beff37321ed868bbcbbdceefe7bad4ab54c5812ec5be5f790d3b4bc536eb8c0"} Jul 02 20:19:37.557674 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:37.557584 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" event={"ID":"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5","Type":"ContainerStarted","Data":"d08655286e5eeaa0c3b7bf86694fb251e9031155054f46a5c4bbd9ed50bd72ba"} Jul 02 20:19:37.557674 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:37.557611 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" event={"ID":"0cff93e5-3bd8-4d6d-a05d-3649be4e11a5","Type":"ContainerStarted","Data":"8de7df7604dff7c576554a82961778c434e88c40893a25766f71adc5b3471376"} Jul 02 20:19:37.573846 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:37.573805 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-shzp9" podStartSLOduration=2.999406848 podStartE2EDuration="4.573790163s" totalImagesPullingTime="1.574383315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:37.571626803 +0000 UTC m=+286.567930036" watchObservedRunningTime="2026-07-02 20:19:37.573790163 +0000 UTC m=+286.570093395" Jul 02 20:19:37.588636 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:37.588596 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-q24zq" podStartSLOduration=2.864484688 podStartE2EDuration="4.588582401s" totalImagesPullingTime="1.724097713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:37.588396415 +0000 UTC m=+286.584699650" watchObservedRunningTime="2026-07-02 20:19:37.588582401 +0000 UTC m=+286.584885633" Jul 02 20:19:37.859826 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:37.859745 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:19:37.862176 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:37.862145 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3386b537-14c7-4ed3-9c7c-7d5d83e7ae93-metrics-tls\") pod \"dns-default-hmr7z\" (UID: \"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93\") " pod="openshift-dns/dns-default-hmr7z" Jul 02 20:19:37.924476 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:37.924440 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:38.138426 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:38.138350 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-fzb4r\"" Jul 02 20:19:38.146584 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:38.146553 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hmr7z" Jul 02 20:19:38.271730 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:38.271703 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:38.282171 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:38.282151 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hmr7z"] Jul 02 20:19:38.284189 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:19:38.284166 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3386b537_14c7_4ed3_9c7c_7d5d83e7ae93.slice/crio-ee10c3a4278f19095136b4044288679c6d50f3db1ce2be7ec9c991def9d85bc1 WatchSource:0}: Error finding container ee10c3a4278f19095136b4044288679c6d50f3db1ce2be7ec9c991def9d85bc1: Status 404 returned error can't find the container with id ee10c3a4278f19095136b4044288679c6d50f3db1ce2be7ec9c991def9d85bc1 Jul 02 20:19:38.561880 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:38.561843 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmr7z" event={"ID":"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93","Type":"ContainerStarted","Data":"ee10c3a4278f19095136b4044288679c6d50f3db1ce2be7ec9c991def9d85bc1"} Jul 02 20:19:39.348954 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:39.348902 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:39.440970 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:39.440891 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:39.619437 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:39.619350 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:39.720892 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:39.720851 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:40.423426 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:40.423400 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:40.524696 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:40.524611 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:40.571392 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:40.571361 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmr7z" event={"ID":"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93","Type":"ContainerStarted","Data":"f5bd2df1f29221145d3cb099759595b5f5592479f15c7078c45a996e56ba0e98"} Jul 02 20:19:40.571530 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:40.571410 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hmr7z" event={"ID":"3386b537-14c7-4ed3-9c7c-7d5d83e7ae93","Type":"ContainerStarted","Data":"735a050ca97294daf90b229c0138ee1610f78bafcf0d5824d50f93b51d58f423"} Jul 02 20:19:40.571530 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:40.571459 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hmr7z" Jul 02 20:19:40.590974 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:40.590924 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hmr7z" podStartSLOduration=251.609735018 podStartE2EDuration="4m13.590906987s" totalImagesPullingTime="1.981171969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:15:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:40.590463103 +0000 UTC m=+289.586766336" watchObservedRunningTime="2026-07-02 20:19:40.590906987 +0000 UTC m=+289.587210219" Jul 02 20:19:42.040437 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.040379 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" podUID="e20f83f3-3574-41f1-99c9-0e22df9a2e0d" containerName="registry" containerID="cri-o://decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89" gracePeriod=30 Jul 02 20:19:42.284903 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.284879 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:19:42.401984 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.401958 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-certificates\") pod \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " Jul 02 20:19:42.402128 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.401992 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4gqt\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-kube-api-access-j4gqt\") pod \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " Jul 02 20:19:42.402128 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.402023 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-image-registry-private-configuration\") pod \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " Jul 02 20:19:42.402128 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.402054 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-ca-trust-extracted\") pod \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " Jul 02 20:19:42.402128 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.402089 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls\") pod \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " Jul 02 20:19:42.402128 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.402107 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-installation-pull-secrets\") pod \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " Jul 02 20:19:42.402390 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.402148 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-bound-sa-token\") pod \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " Jul 02 20:19:42.402390 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.402291 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-trusted-ca\") pod \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\" (UID: \"e20f83f3-3574-41f1-99c9-0e22df9a2e0d\") " Jul 02 20:19:42.402481 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.402452 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e20f83f3-3574-41f1-99c9-0e22df9a2e0d" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:19:42.402788 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.402620 2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-certificates\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:19:42.403079 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.402880 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e20f83f3-3574-41f1-99c9-0e22df9a2e0d" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:19:42.404566 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.404544 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e20f83f3-3574-41f1-99c9-0e22df9a2e0d" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:19:42.404779 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.404754 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e20f83f3-3574-41f1-99c9-0e22df9a2e0d" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:19:42.404850 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.404770 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-kube-api-access-j4gqt" (OuterVolumeSpecName: "kube-api-access-j4gqt") pod "e20f83f3-3574-41f1-99c9-0e22df9a2e0d" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d"). InnerVolumeSpecName "kube-api-access-j4gqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:19:42.404998 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.404978 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e20f83f3-3574-41f1-99c9-0e22df9a2e0d" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:19:42.405103 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.405007 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e20f83f3-3574-41f1-99c9-0e22df9a2e0d" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:19:42.410823 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.410801 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e20f83f3-3574-41f1-99c9-0e22df9a2e0d" (UID: "e20f83f3-3574-41f1-99c9-0e22df9a2e0d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:19:42.503717 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.503684 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4gqt\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-kube-api-access-j4gqt\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:19:42.503717 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.503717 2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-image-registry-private-configuration\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:19:42.503924 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.503731 2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-ca-trust-extracted\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:19:42.503924 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.503744 2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-registry-tls\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:19:42.503924 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.503755 2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-installation-pull-secrets\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:19:42.503924 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.503767 2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-bound-sa-token\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:19:42.503924 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.503778 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e20f83f3-3574-41f1-99c9-0e22df9a2e0d-trusted-ca\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:19:42.579724 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.579682 2565 generic.go:356] "Generic (PLEG): container finished" podID="e20f83f3-3574-41f1-99c9-0e22df9a2e0d" containerID="decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89" exitCode=0 Jul 02 20:19:42.579724 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.579721 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" event={"ID":"e20f83f3-3574-41f1-99c9-0e22df9a2e0d","Type":"ContainerDied","Data":"decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89"} Jul 02 20:19:42.579917 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.579743 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" Jul 02 20:19:42.579917 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.579774 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b5cf7cccf-9vswd" event={"ID":"e20f83f3-3574-41f1-99c9-0e22df9a2e0d","Type":"ContainerDied","Data":"36f1329fa7a5c2865c6379e2fe0659e5b989fe80b569e7e660bd5c493f32c091"} Jul 02 20:19:42.579917 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.579804 2565 scope.go:117] "RemoveContainer" containerID="decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89" Jul 02 20:19:42.588195 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.588179 2565 scope.go:117] "RemoveContainer" containerID="decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89" Jul 02 20:19:42.588480 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:42.588458 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89\": container with ID starting with decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89 not found: ID does not exist" containerID="decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89" Jul 02 20:19:42.588527 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.588490 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89"} err="failed to get container status \"decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89\": rpc error: code = NotFound desc = could not find container \"decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89\": container with ID starting with decd39c0cb01a99f16ab6bd2f57f9dfff348d80718ac22b73dc25a665af8fc89 not found: ID does not exist" Jul 02 20:19:42.600882 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.600857 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7b5cf7cccf-9vswd"] Jul 02 20:19:42.606325 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:42.606304 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7b5cf7cccf-9vswd"] Jul 02 20:19:43.570326 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:43.570296 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e20f83f3-3574-41f1-99c9-0e22df9a2e0d" path="/var/lib/kubelet/pods/e20f83f3-3574-41f1-99c9-0e22df9a2e0d/volumes" Jul 02 20:19:44.565468 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:44.565438 2565 scope.go:117] "RemoveContainer" containerID="294184b6ff5c5657a5552a3ccd5d6a5bcb63d679cdc5aea4a5fd70838a2a912a" Jul 02 20:19:45.370680 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.370645 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-h6w5f"] Jul 02 20:19:45.371039 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.370932 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e20f83f3-3574-41f1-99c9-0e22df9a2e0d" containerName="registry" Jul 02 20:19:45.371039 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.370942 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e20f83f3-3574-41f1-99c9-0e22df9a2e0d" containerName="registry" Jul 02 20:19:45.371039 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.370982 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e20f83f3-3574-41f1-99c9-0e22df9a2e0d" containerName="registry" Jul 02 20:19:45.373770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.373750 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-h6w5f" Jul 02 20:19:45.376441 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.376422 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 02 20:19:45.376548 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.376432 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 02 20:19:45.376846 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.376829 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-l4bqg\"" Jul 02 20:19:45.385231 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.383842 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-h6w5f"] Jul 02 20:19:45.385549 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.385520 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:45.430524 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.430496 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f777\" (UniqueName: \"kubernetes.io/projected/0ff452e7-4b5f-4e94-84b6-245ce46bb584-kube-api-access-8f777\") pod \"downloads-685bfb5f64-h6w5f\" (UID: \"0ff452e7-4b5f-4e94-84b6-245ce46bb584\") " pod="openshift-console/downloads-685bfb5f64-h6w5f" Jul 02 20:19:45.480903 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.480870 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:45.531085 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.531053 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f777\" (UniqueName: \"kubernetes.io/projected/0ff452e7-4b5f-4e94-84b6-245ce46bb584-kube-api-access-8f777\") pod \"downloads-685bfb5f64-h6w5f\" (UID: \"0ff452e7-4b5f-4e94-84b6-245ce46bb584\") " pod="openshift-console/downloads-685bfb5f64-h6w5f" Jul 02 20:19:45.533893 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.533871 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:45.539010 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.538992 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f777\" (UniqueName: \"kubernetes.io/projected/0ff452e7-4b5f-4e94-84b6-245ce46bb584-kube-api-access-8f777\") pod \"downloads-685bfb5f64-h6w5f\" (UID: \"0ff452e7-4b5f-4e94-84b6-245ce46bb584\") " pod="openshift-console/downloads-685bfb5f64-h6w5f" Jul 02 20:19:45.596178 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.596152 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:19:45.596354 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.596264 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" event={"ID":"5ba6abb2-3ddd-4296-9858-6de372075f3a","Type":"ContainerStarted","Data":"4dee832d235fe39dbf48e6ad3952ea7cd7746e5f598bd8bed2deecbdcc7be4b0"} Jul 02 20:19:45.596555 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.596537 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:19:45.601308 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.601291 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" Jul 02 20:19:45.614560 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.614522 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-7b8w7" podStartSLOduration=54.715749196 podStartE2EDuration="57.61450908s" totalImagesPullingTime="2.898759884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:19:45.613814694 +0000 UTC m=+294.610117927" watchObservedRunningTime="2026-07-02 20:19:45.61450908 +0000 UTC m=+294.610812342" Jul 02 20:19:45.681437 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.681407 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:45.688286 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.688265 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-h6w5f" Jul 02 20:19:45.812677 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:45.812624 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-h6w5f"] Jul 02 20:19:45.815940 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:19:45.815910 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff452e7_4b5f_4e94_84b6_245ce46bb584.slice/crio-5a28a62a2aa74a2875730521c674b37b6d7d0dc9ae22eadcd0b7682f382004ed WatchSource:0}: Error finding container 5a28a62a2aa74a2875730521c674b37b6d7d0dc9ae22eadcd0b7682f382004ed: Status 404 returned error can't find the container with id 5a28a62a2aa74a2875730521c674b37b6d7d0dc9ae22eadcd0b7682f382004ed Jul 02 20:19:46.055643 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:46.055556 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:46.156396 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:46.156364 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:46.455355 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:46.455318 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:46.566411 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:19:46.566367 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-hr888" podUID="8caf8e2d-384f-4ecb-a60c-3ce8137a419a" Jul 02 20:19:46.600733 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:46.600696 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-h6w5f" event={"ID":"0ff452e7-4b5f-4e94-84b6-245ce46bb584","Type":"ContainerStarted","Data":"5a28a62a2aa74a2875730521c674b37b6d7d0dc9ae22eadcd0b7682f382004ed"} Jul 02 20:19:46.679502 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:46.679466 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:46.730332 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:46.730203 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:46.879197 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:46.879163 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:46.980634 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:46.980542 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:47.080537 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:47.080510 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:47.259582 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:47.259495 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:47.551632 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:47.551555 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:50.577439 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:50.577405 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hmr7z" Jul 02 20:19:50.591012 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:50.590979 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:50.705903 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:50.705868 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:51.051121 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.051088 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:51.117857 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.117827 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:51.411577 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.411554 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:19:51.411914 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.411892 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:19:51.424668 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.424648 2565 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 20:19:51.687959 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.687875 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8689f67c46-2rxw9"] Jul 02 20:19:51.691365 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.691344 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.694281 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.694121 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 20:19:51.694281 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.694137 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bl9th\"" Jul 02 20:19:51.694281 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.694160 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 20:19:51.695239 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.695200 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 20:19:51.695334 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.695201 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 20:19:51.695543 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.695524 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 20:19:51.700684 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.700664 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8689f67c46-2rxw9"] Jul 02 20:19:51.788170 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.788130 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-serving-cert\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.788383 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.788204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-oauth-serving-cert\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.788383 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.788261 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-console-config\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.788383 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.788316 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpn8f\" (UniqueName: \"kubernetes.io/projected/9c532e4e-5613-4c90-a762-4a89401048ee-kube-api-access-qpn8f\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.788383 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.788378 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-service-ca\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.788576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.788407 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-oauth-config\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.889772 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.889738 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpn8f\" (UniqueName: \"kubernetes.io/projected/9c532e4e-5613-4c90-a762-4a89401048ee-kube-api-access-qpn8f\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.889955 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.889788 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-service-ca\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.889955 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.889910 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-oauth-config\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.890125 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.889955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-serving-cert\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.890125 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.890003 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-oauth-serving-cert\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.890125 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.890033 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-console-config\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.890634 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.890588 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-service-ca\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.890634 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.890631 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-console-config\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.890956 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.890935 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-oauth-serving-cert\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.893293 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.893269 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-oauth-config\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.893395 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.893291 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-serving-cert\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:51.903043 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:51.902997 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpn8f\" (UniqueName: \"kubernetes.io/projected/9c532e4e-5613-4c90-a762-4a89401048ee-kube-api-access-qpn8f\") pod \"console-8689f67c46-2rxw9\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:52.003075 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:52.002996 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:19:52.114742 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:52.114655 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:52.160099 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:52.160071 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8689f67c46-2rxw9"] Jul 02 20:19:52.163718 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:19:52.163691 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c532e4e_5613_4c90_a762_4a89401048ee.slice/crio-3d855141b39416080cd7659d2d4cbc24365e7187e682c63ac89a3a6b9379c79c WatchSource:0}: Error finding container 3d855141b39416080cd7659d2d4cbc24365e7187e682c63ac89a3a6b9379c79c: Status 404 returned error can't find the container with id 3d855141b39416080cd7659d2d4cbc24365e7187e682c63ac89a3a6b9379c79c Jul 02 20:19:52.166885 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:52.166865 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:52.626549 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:52.626508 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8689f67c46-2rxw9" event={"ID":"9c532e4e-5613-4c90-a762-4a89401048ee","Type":"ContainerStarted","Data":"3d855141b39416080cd7659d2d4cbc24365e7187e682c63ac89a3a6b9379c79c"} Jul 02 20:19:55.690129 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:55.690092 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:55.787765 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:55.787721 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:58.279313 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:58.279284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:58.336031 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:58.335977 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:59.676958 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:59.676915 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:19:59.775079 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:19:59.775040 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:00.565381 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:00.565347 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:20:00.887181 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:00.887104 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:00.946716 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:00.946667 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:03.675397 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:03.675278 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-h6w5f" event={"ID":"0ff452e7-4b5f-4e94-84b6-245ce46bb584","Type":"ContainerStarted","Data":"4d4ebff54bb3aec1870a1be0e9c524e6e3f70032040c3f43ab966e3dab842293"} Jul 02 20:20:03.675397 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:03.675358 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-h6w5f" Jul 02 20:20:03.677428 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:03.677396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8689f67c46-2rxw9" event={"ID":"9c532e4e-5613-4c90-a762-4a89401048ee","Type":"ContainerStarted","Data":"30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e"} Jul 02 20:20:03.686923 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:03.686899 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-h6w5f" Jul 02 20:20:03.693969 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:03.693917 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-h6w5f" podStartSLOduration=1.650275099 podStartE2EDuration="18.693901372s" totalImagesPullingTime="17.043626273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:20:03.691491917 +0000 UTC m=+312.687795151" watchObservedRunningTime="2026-07-02 20:20:03.693901372 +0000 UTC m=+312.690204607" Jul 02 20:20:03.723494 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:03.723440 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8689f67c46-2rxw9" podStartSLOduration=2.069219936 podStartE2EDuration="12.723421815s" totalImagesPullingTime="10.654201879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:19:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:20:03.721528288 +0000 UTC m=+312.717831533" watchObservedRunningTime="2026-07-02 20:20:03.723421815 +0000 UTC m=+312.719725054" Jul 02 20:20:06.033379 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:06.033339 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:06.133692 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:06.133648 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:06.232354 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:06.232322 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:08.703203 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:08.703172 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:08.753993 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:08.753962 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:12.003639 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:12.003591 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:20:12.003639 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:12.003647 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:20:12.010153 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:12.010133 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:20:12.717818 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:12.717787 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:20:13.519606 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:13.519570 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8689f67c46-2rxw9"] Jul 02 20:20:16.284965 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:16.284930 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:16.336736 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:16.336710 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:18.736662 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:18.736629 2565 generic.go:356] "Generic (PLEG): container finished" podID="6e15797b-e921-4425-b53f-94b801477686" containerID="5988a3806dba30591f40243aab07df8a090b799ccd5739a9fd81b42824ed3bd7" exitCode=0 Jul 02 20:20:18.737158 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:18.736701 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" event={"ID":"6e15797b-e921-4425-b53f-94b801477686","Type":"ContainerDied","Data":"5988a3806dba30591f40243aab07df8a090b799ccd5739a9fd81b42824ed3bd7"} Jul 02 20:20:18.737158 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:18.737058 2565 scope.go:117] "RemoveContainer" containerID="5988a3806dba30591f40243aab07df8a090b799ccd5739a9fd81b42824ed3bd7" Jul 02 20:20:19.742434 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:19.742402 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-7r8wz" event={"ID":"6e15797b-e921-4425-b53f-94b801477686","Type":"ContainerStarted","Data":"aa52237ad046e936879cd40fce4be3c84a79315285118e1b9ebd2072bf777508"} Jul 02 20:20:21.041926 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:21.041892 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:21.103841 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:21.103807 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:36.037864 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:36.037828 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:36.138115 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:36.138080 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:39.740926 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:39.740861 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-8689f67c46-2rxw9" podUID="9c532e4e-5613-4c90-a762-4a89401048ee" containerName="console" containerID="cri-o://30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e" gracePeriod=15 Jul 02 20:20:40.010342 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.010285 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:20:40.140403 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.140375 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-oauth-config\") pod \"9c532e4e-5613-4c90-a762-4a89401048ee\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " Jul 02 20:20:40.140593 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.140424 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-oauth-serving-cert\") pod \"9c532e4e-5613-4c90-a762-4a89401048ee\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " Jul 02 20:20:40.140593 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.140462 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-console-config\") pod \"9c532e4e-5613-4c90-a762-4a89401048ee\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " Jul 02 20:20:40.140593 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.140477 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-service-ca\") pod \"9c532e4e-5613-4c90-a762-4a89401048ee\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " Jul 02 20:20:40.140593 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.140513 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpn8f\" (UniqueName: \"kubernetes.io/projected/9c532e4e-5613-4c90-a762-4a89401048ee-kube-api-access-qpn8f\") pod \"9c532e4e-5613-4c90-a762-4a89401048ee\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " Jul 02 20:20:40.140593 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.140531 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-serving-cert\") pod \"9c532e4e-5613-4c90-a762-4a89401048ee\" (UID: \"9c532e4e-5613-4c90-a762-4a89401048ee\") " Jul 02 20:20:40.140911 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.140885 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9c532e4e-5613-4c90-a762-4a89401048ee" (UID: "9c532e4e-5613-4c90-a762-4a89401048ee"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:40.140975 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.140901 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-console-config" (OuterVolumeSpecName: "console-config") pod "9c532e4e-5613-4c90-a762-4a89401048ee" (UID: "9c532e4e-5613-4c90-a762-4a89401048ee"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:40.140975 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.140910 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-service-ca" (OuterVolumeSpecName: "service-ca") pod "9c532e4e-5613-4c90-a762-4a89401048ee" (UID: "9c532e4e-5613-4c90-a762-4a89401048ee"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:20:40.142999 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.142971 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9c532e4e-5613-4c90-a762-4a89401048ee" (UID: "9c532e4e-5613-4c90-a762-4a89401048ee"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:40.142999 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.142987 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9c532e4e-5613-4c90-a762-4a89401048ee" (UID: "9c532e4e-5613-4c90-a762-4a89401048ee"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:20:40.143139 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.142972 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c532e4e-5613-4c90-a762-4a89401048ee-kube-api-access-qpn8f" (OuterVolumeSpecName: "kube-api-access-qpn8f") pod "9c532e4e-5613-4c90-a762-4a89401048ee" (UID: "9c532e4e-5613-4c90-a762-4a89401048ee"). InnerVolumeSpecName "kube-api-access-qpn8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:20:40.241715 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.241680 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpn8f\" (UniqueName: \"kubernetes.io/projected/9c532e4e-5613-4c90-a762-4a89401048ee-kube-api-access-qpn8f\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:20:40.241715 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.241708 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-serving-cert\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:20:40.241715 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.241717 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9c532e4e-5613-4c90-a762-4a89401048ee-console-oauth-config\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:20:40.241931 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.241726 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-oauth-serving-cert\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:20:40.241931 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.241735 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-console-config\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:20:40.241931 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.241743 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9c532e4e-5613-4c90-a762-4a89401048ee-service-ca\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:20:40.820317 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.820285 2565 generic.go:356] "Generic (PLEG): container finished" podID="9c532e4e-5613-4c90-a762-4a89401048ee" containerID="30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e" exitCode=0 Jul 02 20:20:40.820742 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.820329 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8689f67c46-2rxw9" Jul 02 20:20:40.820742 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.820347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8689f67c46-2rxw9" event={"ID":"9c532e4e-5613-4c90-a762-4a89401048ee","Type":"ContainerDied","Data":"30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e"} Jul 02 20:20:40.820742 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.820386 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8689f67c46-2rxw9" event={"ID":"9c532e4e-5613-4c90-a762-4a89401048ee","Type":"ContainerDied","Data":"3d855141b39416080cd7659d2d4cbc24365e7187e682c63ac89a3a6b9379c79c"} Jul 02 20:20:40.820742 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.820405 2565 scope.go:117] "RemoveContainer" containerID="30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e" Jul 02 20:20:40.829240 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.829195 2565 scope.go:117] "RemoveContainer" containerID="30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e" Jul 02 20:20:40.829489 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:20:40.829469 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e\": container with ID starting with 30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e not found: ID does not exist" containerID="30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e" Jul 02 20:20:40.829567 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.829499 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e"} err="failed to get container status \"30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e\": rpc error: code = NotFound desc = could not find container \"30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e\": container with ID starting with 30aab6b4e1b82553fe3418be58f0168b82f838ebc49c2180478f4a18346dfe8e not found: ID does not exist" Jul 02 20:20:40.842046 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.842017 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8689f67c46-2rxw9"] Jul 02 20:20:40.845919 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:40.845899 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8689f67c46-2rxw9"] Jul 02 20:20:41.571434 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:41.571365 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c532e4e-5613-4c90-a762-4a89401048ee" path="/var/lib/kubelet/pods/9c532e4e-5613-4c90-a762-4a89401048ee/volumes" Jul 02 20:20:48.723731 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:48.723699 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:48.779247 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:48.779193 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:52.312457 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:52.312424 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:52.806527 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:52.806498 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:20:57.401750 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.401715 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6cdfd6464f-wg729"] Jul 02 20:20:57.402788 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.402764 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c532e4e-5613-4c90-a762-4a89401048ee" containerName="console" Jul 02 20:20:57.402968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.402958 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c532e4e-5613-4c90-a762-4a89401048ee" containerName="console" Jul 02 20:20:57.403153 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.403142 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c532e4e-5613-4c90-a762-4a89401048ee" containerName="console" Jul 02 20:20:57.406298 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.406275 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.409150 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.409128 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 02 20:20:57.409290 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.409176 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 02 20:20:57.409290 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.409237 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 02 20:20:57.409503 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.409484 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 02 20:20:57.409908 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.409888 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 02 20:20:57.411098 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.411079 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-889b5\"" Jul 02 20:20:57.414472 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.414453 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 02 20:20:57.418272 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.418253 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cdfd6464f-wg729"] Jul 02 20:20:57.479286 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.479247 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bb49ec6-6170-441f-8698-180685fd280d-metrics-client-ca\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.479454 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.479303 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-telemeter-client-tls\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.479454 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.479335 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7wt\" (UniqueName: \"kubernetes.io/projected/2bb49ec6-6170-441f-8698-180685fd280d-kube-api-access-8r7wt\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.479454 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.479366 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb49ec6-6170-441f-8698-180685fd280d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.479454 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.479392 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-federate-client-tls\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.479454 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.479414 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-secret-telemeter-client\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.479454 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.479430 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.479641 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.479523 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb49ec6-6170-441f-8698-180685fd280d-serving-certs-ca-bundle\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.580670 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.580639 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-telemeter-client-tls\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.580868 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.580678 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7wt\" (UniqueName: \"kubernetes.io/projected/2bb49ec6-6170-441f-8698-180685fd280d-kube-api-access-8r7wt\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.580868 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.580706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb49ec6-6170-441f-8698-180685fd280d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.580868 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.580842 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-federate-client-tls\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.581033 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.580886 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-secret-telemeter-client\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.581033 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.580919 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.581033 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.581010 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb49ec6-6170-441f-8698-180685fd280d-serving-certs-ca-bundle\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.581175 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.581039 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bb49ec6-6170-441f-8698-180685fd280d-metrics-client-ca\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.581914 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.581886 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb49ec6-6170-441f-8698-180685fd280d-serving-certs-ca-bundle\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.582060 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.581959 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bb49ec6-6170-441f-8698-180685fd280d-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.582557 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.582533 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bb49ec6-6170-441f-8698-180685fd280d-metrics-client-ca\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.583641 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.583598 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-secret-telemeter-client\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.583748 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.583709 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-federate-client-tls\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.584396 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.584007 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-telemeter-client-tls\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.584396 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.584363 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bb49ec6-6170-441f-8698-180685fd280d-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.588891 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.588859 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7wt\" (UniqueName: \"kubernetes.io/projected/2bb49ec6-6170-441f-8698-180685fd280d-kube-api-access-8r7wt\") pod \"telemeter-client-6cdfd6464f-wg729\" (UID: \"2bb49ec6-6170-441f-8698-180685fd280d\") " pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.718837 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.718750 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" Jul 02 20:20:57.871310 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.871279 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6cdfd6464f-wg729"] Jul 02 20:20:57.874672 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:20:57.874645 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb49ec6_6170_441f_8698_180685fd280d.slice/crio-be86760f737f1496d8fc10e6745d4c293a49b6957742212ab0b003fa2b2c9d45 WatchSource:0}: Error finding container be86760f737f1496d8fc10e6745d4c293a49b6957742212ab0b003fa2b2c9d45: Status 404 returned error can't find the container with id be86760f737f1496d8fc10e6745d4c293a49b6957742212ab0b003fa2b2c9d45 Jul 02 20:20:57.876986 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.876970 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:20:57.881915 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:20:57.881885 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" event={"ID":"2bb49ec6-6170-441f-8698-180685fd280d","Type":"ContainerStarted","Data":"be86760f737f1496d8fc10e6745d4c293a49b6957742212ab0b003fa2b2c9d45"} Jul 02 20:21:00.899638 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:00.899594 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" event={"ID":"2bb49ec6-6170-441f-8698-180685fd280d","Type":"ContainerStarted","Data":"6b64132a305eb2166df116a24cca9685c96ee79796844a0512c3ee125d483817"} Jul 02 20:21:01.905795 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:01.905758 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" event={"ID":"2bb49ec6-6170-441f-8698-180685fd280d","Type":"ContainerStarted","Data":"30728d6046c5887fa1f69e9a39694688d926176b351e93107f4793e236f12c81"} Jul 02 20:21:01.906232 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:01.905822 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" event={"ID":"2bb49ec6-6170-441f-8698-180685fd280d","Type":"ContainerStarted","Data":"5cfa08cd743c0bd2b8329d41b9cdb985ca8d3e3bf467d13b2359bf9ad847ce91"} Jul 02 20:21:01.927726 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:01.927676 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6cdfd6464f-wg729" podStartSLOduration=1.492351869 podStartE2EDuration="4.927662496s" totalImagesPullingTime="3.435310627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:20:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:21:01.925533402 +0000 UTC m=+370.921836634" watchObservedRunningTime="2026-07-02 20:21:01.927662496 +0000 UTC m=+370.923965729" Jul 02 20:21:02.643572 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.643537 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6878995d9f-w57cj"] Jul 02 20:21:02.646361 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.646333 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.650231 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.649280 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 20:21:02.650231 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.649661 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 20:21:02.650231 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.649837 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 20:21:02.650670 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.650653 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bl9th\"" Jul 02 20:21:02.651182 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.651159 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 20:21:02.651886 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.651864 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 20:21:02.656280 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.656257 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 02 20:21:02.658655 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.658598 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6878995d9f-w57cj"] Jul 02 20:21:02.716800 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.716770 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:21:02.725029 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.725002 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-service-ca\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.725278 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.725260 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-oauth-serving-cert\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.725455 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.725434 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-oauth-config\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.725558 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.725466 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-trusted-ca-bundle\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.725558 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.725510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-serving-cert\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.725642 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.725573 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbjv\" (UniqueName: \"kubernetes.io/projected/b6b15451-0b9a-4bfe-aa59-3776658a50f5-kube-api-access-tnbjv\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.725642 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.725597 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-config\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.767893 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.767864 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:21:02.826568 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.826530 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-serving-cert\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.826568 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.826573 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbjv\" (UniqueName: \"kubernetes.io/projected/b6b15451-0b9a-4bfe-aa59-3776658a50f5-kube-api-access-tnbjv\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.826806 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.826589 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-config\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.826806 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.826784 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-service-ca\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.826911 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.826821 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-oauth-serving-cert\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.826911 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.826853 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-oauth-config\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.826911 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.826877 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-trusted-ca-bundle\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.827425 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.827396 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-config\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.827616 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.827586 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-oauth-serving-cert\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.827616 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.827488 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-service-ca\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.827762 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.827689 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-trusted-ca-bundle\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.829972 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.829947 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-serving-cert\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.830363 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.830349 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-oauth-config\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.834809 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.834765 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbjv\" (UniqueName: \"kubernetes.io/projected/b6b15451-0b9a-4bfe-aa59-3776658a50f5-kube-api-access-tnbjv\") pod \"console-6878995d9f-w57cj\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:02.958560 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:02.958528 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:03.083434 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:03.083409 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6878995d9f-w57cj"] Jul 02 20:21:03.085884 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:21:03.085859 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b15451_0b9a_4bfe_aa59_3776658a50f5.slice/crio-5f112a19fb5c5cea619bd7ee6b06022a9b47711ba22d70fe479922f6e03deb9f WatchSource:0}: Error finding container 5f112a19fb5c5cea619bd7ee6b06022a9b47711ba22d70fe479922f6e03deb9f: Status 404 returned error can't find the container with id 5f112a19fb5c5cea619bd7ee6b06022a9b47711ba22d70fe479922f6e03deb9f Jul 02 20:21:03.914323 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:03.914286 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878995d9f-w57cj" event={"ID":"b6b15451-0b9a-4bfe-aa59-3776658a50f5","Type":"ContainerStarted","Data":"5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58"} Jul 02 20:21:03.914485 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:03.914377 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878995d9f-w57cj" event={"ID":"b6b15451-0b9a-4bfe-aa59-3776658a50f5","Type":"ContainerStarted","Data":"5f112a19fb5c5cea619bd7ee6b06022a9b47711ba22d70fe479922f6e03deb9f"} Jul 02 20:21:03.930923 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:03.930877 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6878995d9f-w57cj" podStartSLOduration=1.930864527 podStartE2EDuration="1.930864527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:21:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:21:03.929915537 +0000 UTC m=+372.926218771" watchObservedRunningTime="2026-07-02 20:21:03.930864527 +0000 UTC m=+372.927167760" Jul 02 20:21:05.348165 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:05.348129 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:21:05.350632 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:05.350609 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8caf8e2d-384f-4ecb-a60c-3ce8137a419a-metrics-certs\") pod \"network-metrics-daemon-hr888\" (UID: \"8caf8e2d-384f-4ecb-a60c-3ce8137a419a\") " pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:21:05.369283 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:05.369262 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2jpl6\"" Jul 02 20:21:05.377298 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:05.377280 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hr888" Jul 02 20:21:05.501510 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:05.501484 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hr888"] Jul 02 20:21:05.504132 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:21:05.504107 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8caf8e2d_384f_4ecb_a60c_3ce8137a419a.slice/crio-d1fe6276dbe050e5484803f162780bf3b9420bc84ffa1675f743b1b5ce407ce0 WatchSource:0}: Error finding container d1fe6276dbe050e5484803f162780bf3b9420bc84ffa1675f743b1b5ce407ce0: Status 404 returned error can't find the container with id d1fe6276dbe050e5484803f162780bf3b9420bc84ffa1675f743b1b5ce407ce0 Jul 02 20:21:05.922013 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:05.921983 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hr888" event={"ID":"8caf8e2d-384f-4ecb-a60c-3ce8137a419a","Type":"ContainerStarted","Data":"d1fe6276dbe050e5484803f162780bf3b9420bc84ffa1675f743b1b5ce407ce0"} Jul 02 20:21:06.927378 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:06.927349 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hr888" event={"ID":"8caf8e2d-384f-4ecb-a60c-3ce8137a419a","Type":"ContainerStarted","Data":"0a38d6785ceb02a95e9f53b1b3f618c022c5fcc03f6fff31161b673c5442a596"} Jul 02 20:21:06.927733 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:06.927396 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hr888" event={"ID":"8caf8e2d-384f-4ecb-a60c-3ce8137a419a","Type":"ContainerStarted","Data":"fa6b8a569fe330a1503a6303ca76209e7e91e20c3385e330cbe8ecd4b830c973"} Jul 02 20:21:06.942438 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:06.942396 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hr888" podStartSLOduration=373.727003612 podStartE2EDuration="6m14.942378932s" totalImagesPullingTime="1.21537532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:14:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:21:06.941875996 +0000 UTC m=+375.938179230" watchObservedRunningTime="2026-07-02 20:21:06.942378932 +0000 UTC m=+375.938682165" Jul 02 20:21:07.024133 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:07.024058 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:21:07.081339 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:07.081315 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:21:07.717202 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:07.717175 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:21:07.772376 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:07.772340 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:21:12.959229 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:12.959183 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:12.959605 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:12.959252 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:12.964015 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:12.963991 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:13.956869 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:13.956838 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:21:14.104448 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:14.104422 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:21:14.209652 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:14.209550 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:21:14.259843 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:14.259811 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:21:49.473746 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:49.473714 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:21:49.525371 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:21:49.525345 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:07.602005 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:07.601970 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:07.656928 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:07.656893 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:13.380706 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:13.380628 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:19.422091 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:19.422059 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:19.476581 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:19.476549 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:36.080921 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:36.080888 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:36.173264 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:36.173229 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:45.384923 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.384891 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c7c85466b-56gk7"] Jul 02 20:22:45.388094 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.388068 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.402956 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.402925 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7c85466b-56gk7"] Jul 02 20:22:45.467118 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.467076 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-config\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.467364 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.467137 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68b8\" (UniqueName: \"kubernetes.io/projected/d8130ea0-86fb-4b36-b746-cc0129a91f1a-kube-api-access-z68b8\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.467364 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.467194 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-service-ca\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.467364 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.467273 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-serving-cert\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.467364 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.467298 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-trusted-ca-bundle\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.467364 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.467320 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-oauth-config\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.467364 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.467345 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-oauth-serving-cert\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.469148 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.469127 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:45.523253 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.523202 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:45.567720 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.567667 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68b8\" (UniqueName: \"kubernetes.io/projected/d8130ea0-86fb-4b36-b746-cc0129a91f1a-kube-api-access-z68b8\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.567720 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.567716 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-service-ca\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.567932 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.567733 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-serving-cert\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.567932 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.567849 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-trusted-ca-bundle\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.567932 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.567887 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-oauth-config\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.567932 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.567915 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-oauth-serving-cert\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.568174 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.568001 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-config\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.568821 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.568793 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-service-ca\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.568965 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.568822 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-config\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.568965 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.568861 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-oauth-serving-cert\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.569087 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.569037 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-trusted-ca-bundle\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.570406 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.570386 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-oauth-config\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.570685 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.570666 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-serving-cert\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.575334 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.575316 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68b8\" (UniqueName: \"kubernetes.io/projected/d8130ea0-86fb-4b36-b746-cc0129a91f1a-kube-api-access-z68b8\") pod \"console-6c7c85466b-56gk7\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.579072 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.579053 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:45.640073 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.639995 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:45.703834 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.703797 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:45.842005 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:45.841975 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c7c85466b-56gk7"] Jul 02 20:22:45.844989 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:22:45.844964 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8130ea0_86fb_4b36_b746_cc0129a91f1a.slice/crio-cdc65f68186c5aec851e89461a39becca9a2bca8416926eb2872b2dc30e479f0 WatchSource:0}: Error finding container cdc65f68186c5aec851e89461a39becca9a2bca8416926eb2872b2dc30e479f0: Status 404 returned error can't find the container with id cdc65f68186c5aec851e89461a39becca9a2bca8416926eb2872b2dc30e479f0 Jul 02 20:22:46.302559 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:46.302466 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7c85466b-56gk7" event={"ID":"d8130ea0-86fb-4b36-b746-cc0129a91f1a","Type":"ContainerStarted","Data":"7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859"} Jul 02 20:22:46.302559 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:46.302524 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7c85466b-56gk7" event={"ID":"d8130ea0-86fb-4b36-b746-cc0129a91f1a","Type":"ContainerStarted","Data":"cdc65f68186c5aec851e89461a39becca9a2bca8416926eb2872b2dc30e479f0"} Jul 02 20:22:46.328393 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:46.328348 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c7c85466b-56gk7" podStartSLOduration=1.328333775 podStartE2EDuration="1.328333775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:22:46.328001552 +0000 UTC m=+475.324304786" watchObservedRunningTime="2026-07-02 20:22:46.328333775 +0000 UTC m=+475.324637007" Jul 02 20:22:46.425040 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:46.425010 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:46.527822 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:46.527792 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:55.704925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:55.704888 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:55.705332 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:55.704951 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:55.709516 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:55.709495 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:56.346257 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:56.346229 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:22:56.396089 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:56.396055 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6878995d9f-w57cj"] Jul 02 20:22:56.538384 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:56.538356 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:56.641798 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:56.641724 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:22:56.711572 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:22:56.711542 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:15.910141 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:15.910111 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:20.669128 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:20.669097 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:20.815515 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:20.815485 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:21.417578 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.417520 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6878995d9f-w57cj" podUID="b6b15451-0b9a-4bfe-aa59-3776658a50f5" containerName="console" containerID="cri-o://5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58" gracePeriod=15 Jul 02 20:23:21.673952 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.673898 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:23:21.763063 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763025 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-trusted-ca-bundle\") pod \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " Jul 02 20:23:21.763275 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763099 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-service-ca\") pod \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " Jul 02 20:23:21.763275 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763118 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-serving-cert\") pod \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " Jul 02 20:23:21.763275 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763153 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-config\") pod \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " Jul 02 20:23:21.763275 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763180 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-oauth-config\") pod \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " Jul 02 20:23:21.763481 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763356 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnbjv\" (UniqueName: \"kubernetes.io/projected/b6b15451-0b9a-4bfe-aa59-3776658a50f5-kube-api-access-tnbjv\") pod \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " Jul 02 20:23:21.763481 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763402 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-oauth-serving-cert\") pod \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\" (UID: \"b6b15451-0b9a-4bfe-aa59-3776658a50f5\") " Jul 02 20:23:21.763579 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763504 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b6b15451-0b9a-4bfe-aa59-3776658a50f5" (UID: "b6b15451-0b9a-4bfe-aa59-3776658a50f5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:23:21.763632 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763613 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-service-ca" (OuterVolumeSpecName: "service-ca") pod "b6b15451-0b9a-4bfe-aa59-3776658a50f5" (UID: "b6b15451-0b9a-4bfe-aa59-3776658a50f5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:23:21.763767 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763741 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-config" (OuterVolumeSpecName: "console-config") pod "b6b15451-0b9a-4bfe-aa59-3776658a50f5" (UID: "b6b15451-0b9a-4bfe-aa59-3776658a50f5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:23:21.763849 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763793 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-service-ca\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:23:21.763849 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763814 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-trusted-ca-bundle\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:23:21.763849 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.763815 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b6b15451-0b9a-4bfe-aa59-3776658a50f5" (UID: "b6b15451-0b9a-4bfe-aa59-3776658a50f5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:23:21.765573 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.765543 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b15451-0b9a-4bfe-aa59-3776658a50f5-kube-api-access-tnbjv" (OuterVolumeSpecName: "kube-api-access-tnbjv") pod "b6b15451-0b9a-4bfe-aa59-3776658a50f5" (UID: "b6b15451-0b9a-4bfe-aa59-3776658a50f5"). InnerVolumeSpecName "kube-api-access-tnbjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:23:21.765967 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.765948 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b6b15451-0b9a-4bfe-aa59-3776658a50f5" (UID: "b6b15451-0b9a-4bfe-aa59-3776658a50f5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:23:21.766015 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.765962 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b6b15451-0b9a-4bfe-aa59-3776658a50f5" (UID: "b6b15451-0b9a-4bfe-aa59-3776658a50f5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:23:21.865198 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.865167 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-serving-cert\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:23:21.865198 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.865193 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-config\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:23:21.865198 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.865203 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b6b15451-0b9a-4bfe-aa59-3776658a50f5-console-oauth-config\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:23:21.865440 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.865232 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnbjv\" (UniqueName: \"kubernetes.io/projected/b6b15451-0b9a-4bfe-aa59-3776658a50f5-kube-api-access-tnbjv\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:23:21.865440 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:21.865241 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b6b15451-0b9a-4bfe-aa59-3776658a50f5-oauth-serving-cert\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:23:22.442892 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:22.442864 2565 generic.go:356] "Generic (PLEG): container finished" podID="b6b15451-0b9a-4bfe-aa59-3776658a50f5" containerID="5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58" exitCode=0 Jul 02 20:23:22.443065 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:22.442906 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878995d9f-w57cj" event={"ID":"b6b15451-0b9a-4bfe-aa59-3776658a50f5","Type":"ContainerDied","Data":"5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58"} Jul 02 20:23:22.443065 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:22.442910 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6878995d9f-w57cj" Jul 02 20:23:22.443065 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:22.442945 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6878995d9f-w57cj" event={"ID":"b6b15451-0b9a-4bfe-aa59-3776658a50f5","Type":"ContainerDied","Data":"5f112a19fb5c5cea619bd7ee6b06022a9b47711ba22d70fe479922f6e03deb9f"} Jul 02 20:23:22.443065 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:22.442965 2565 scope.go:117] "RemoveContainer" containerID="5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58" Jul 02 20:23:22.452943 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:22.452923 2565 scope.go:117] "RemoveContainer" containerID="5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58" Jul 02 20:23:22.453199 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:23:22.453180 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58\": container with ID starting with 5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58 not found: ID does not exist" containerID="5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58" Jul 02 20:23:22.453293 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:22.453228 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58"} err="failed to get container status \"5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58\": rpc error: code = NotFound desc = could not find container \"5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58\": container with ID starting with 5a6eebcc702a5b43b9d670ac3308086512fa122a943399c31574a7a1b2011b58 not found: ID does not exist" Jul 02 20:23:22.464250 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:22.464229 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6878995d9f-w57cj"] Jul 02 20:23:22.468292 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:22.468272 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6878995d9f-w57cj"] Jul 02 20:23:23.570100 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:23.570068 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6b15451-0b9a-4bfe-aa59-3776658a50f5" path="/var/lib/kubelet/pods/b6b15451-0b9a-4bfe-aa59-3776658a50f5/volumes" Jul 02 20:23:47.316410 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:47.316325 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:48.690933 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:48.690900 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:51.125294 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:51.125262 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:51.183322 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:51.183296 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:51.719587 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:51.719501 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:51.857769 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:51.857736 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:52.809851 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:52.809817 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:52.913089 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:52.913056 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:54.813641 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:54.813612 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:55.201526 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.201495 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:55.523028 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.522952 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xshp2"] Jul 02 20:23:55.523285 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.523272 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6b15451-0b9a-4bfe-aa59-3776658a50f5" containerName="console" Jul 02 20:23:55.523336 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.523287 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6b15451-0b9a-4bfe-aa59-3776658a50f5" containerName="console" Jul 02 20:23:55.523336 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.523333 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6b15451-0b9a-4bfe-aa59-3776658a50f5" containerName="console" Jul 02 20:23:55.527415 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.527396 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xshp2" Jul 02 20:23:55.529967 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.529944 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 02 20:23:55.531035 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.531014 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 02 20:23:55.531154 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.531037 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-h7q65\"" Jul 02 20:23:55.535738 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.535712 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xshp2"] Jul 02 20:23:55.634335 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.634301 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz6tc\" (UniqueName: \"kubernetes.io/projected/6eb12092-5e71-49d5-a92a-fcba29f1b0ca-kube-api-access-tz6tc\") pod \"cert-manager-cainjector-559cdd8c87-xshp2\" (UID: \"6eb12092-5e71-49d5-a92a-fcba29f1b0ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xshp2" Jul 02 20:23:55.634508 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.634413 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6eb12092-5e71-49d5-a92a-fcba29f1b0ca-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xshp2\" (UID: \"6eb12092-5e71-49d5-a92a-fcba29f1b0ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xshp2" Jul 02 20:23:55.735622 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.735592 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6eb12092-5e71-49d5-a92a-fcba29f1b0ca-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xshp2\" (UID: \"6eb12092-5e71-49d5-a92a-fcba29f1b0ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xshp2" Jul 02 20:23:55.735782 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.735636 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6tc\" (UniqueName: \"kubernetes.io/projected/6eb12092-5e71-49d5-a92a-fcba29f1b0ca-kube-api-access-tz6tc\") pod \"cert-manager-cainjector-559cdd8c87-xshp2\" (UID: \"6eb12092-5e71-49d5-a92a-fcba29f1b0ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xshp2" Jul 02 20:23:55.744020 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.743988 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6eb12092-5e71-49d5-a92a-fcba29f1b0ca-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-xshp2\" (UID: \"6eb12092-5e71-49d5-a92a-fcba29f1b0ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xshp2" Jul 02 20:23:55.744547 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.744525 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6tc\" (UniqueName: \"kubernetes.io/projected/6eb12092-5e71-49d5-a92a-fcba29f1b0ca-kube-api-access-tz6tc\") pod \"cert-manager-cainjector-559cdd8c87-xshp2\" (UID: \"6eb12092-5e71-49d5-a92a-fcba29f1b0ca\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-xshp2" Jul 02 20:23:55.852047 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.851954 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xshp2" Jul 02 20:23:55.980434 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:55.980410 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-xshp2"] Jul 02 20:23:55.982956 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:23:55.982925 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6eb12092_5e71_49d5_a92a_fcba29f1b0ca.slice/crio-0f052feb76b855c217dabba621cc74d92aee8ecf36d027ae8998ffbb642efb39 WatchSource:0}: Error finding container 0f052feb76b855c217dabba621cc74d92aee8ecf36d027ae8998ffbb642efb39: Status 404 returned error can't find the container with id 0f052feb76b855c217dabba621cc74d92aee8ecf36d027ae8998ffbb642efb39 Jul 02 20:23:56.574048 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:56.574013 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xshp2" event={"ID":"6eb12092-5e71-49d5-a92a-fcba29f1b0ca","Type":"ContainerStarted","Data":"0f052feb76b855c217dabba621cc74d92aee8ecf36d027ae8998ffbb642efb39"} Jul 02 20:23:58.327389 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:58.327356 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:58.428527 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:58.428496 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:23:59.588203 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:59.588168 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xshp2" event={"ID":"6eb12092-5e71-49d5-a92a-fcba29f1b0ca","Type":"ContainerStarted","Data":"27e68e094a39111ab24e11cc1a2aedb3c859e83963123b9837f31afaf253bc9d"} Jul 02 20:23:59.603656 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:23:59.603569 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-xshp2" podStartSLOduration=1.514203341 podStartE2EDuration="4.603547275s" totalImagesPullingTime="3.089343934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:23:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:23:59.60337769 +0000 UTC m=+548.599680928" watchObservedRunningTime="2026-07-02 20:23:59.603547275 +0000 UTC m=+548.599850508" Jul 02 20:24:01.245634 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:01.245605 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:01.346226 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:01.346177 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:03.130657 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:03.130627 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:03.230041 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:03.230004 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:03.407371 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:03.407337 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:03.717582 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:03.717495 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:05.752104 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.752075 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6"] Jul 02 20:24:05.755380 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.755366 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6" Jul 02 20:24:05.757908 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.757883 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 02 20:24:05.757908 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.757883 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 02 20:24:05.758838 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.758820 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-59vdc\"" Jul 02 20:24:05.762389 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.762071 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6"] Jul 02 20:24:05.819954 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.819918 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c28875e4-241d-4f91-a517-83d78a5c4dd6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7fvt6\" (UID: \"c28875e4-241d-4f91-a517-83d78a5c4dd6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6" Jul 02 20:24:05.820133 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.819995 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4qbr\" (UniqueName: \"kubernetes.io/projected/c28875e4-241d-4f91-a517-83d78a5c4dd6-kube-api-access-w4qbr\") pod \"openshift-lws-operator-fd8ccff4c-7fvt6\" (UID: \"c28875e4-241d-4f91-a517-83d78a5c4dd6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6" Jul 02 20:24:05.921031 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.920997 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c28875e4-241d-4f91-a517-83d78a5c4dd6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7fvt6\" (UID: \"c28875e4-241d-4f91-a517-83d78a5c4dd6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6" Jul 02 20:24:05.921289 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.921058 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4qbr\" (UniqueName: \"kubernetes.io/projected/c28875e4-241d-4f91-a517-83d78a5c4dd6-kube-api-access-w4qbr\") pod \"openshift-lws-operator-fd8ccff4c-7fvt6\" (UID: \"c28875e4-241d-4f91-a517-83d78a5c4dd6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6" Jul 02 20:24:05.921466 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.921444 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c28875e4-241d-4f91-a517-83d78a5c4dd6-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7fvt6\" (UID: \"c28875e4-241d-4f91-a517-83d78a5c4dd6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6" Jul 02 20:24:05.929193 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:05.929166 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4qbr\" (UniqueName: \"kubernetes.io/projected/c28875e4-241d-4f91-a517-83d78a5c4dd6-kube-api-access-w4qbr\") pod \"openshift-lws-operator-fd8ccff4c-7fvt6\" (UID: \"c28875e4-241d-4f91-a517-83d78a5c4dd6\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6" Jul 02 20:24:06.010572 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:06.010474 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:06.064565 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:06.064530 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6" Jul 02 20:24:06.110651 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:06.110613 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:06.194500 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:06.194478 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6"] Jul 02 20:24:06.197202 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:24:06.197171 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc28875e4_241d_4f91_a517_83d78a5c4dd6.slice/crio-7ca3da6a17be64c9afc3c6e010d80e8d09062cd8ab1e1144f9c5dae8291e9cc0 WatchSource:0}: Error finding container 7ca3da6a17be64c9afc3c6e010d80e8d09062cd8ab1e1144f9c5dae8291e9cc0: Status 404 returned error can't find the container with id 7ca3da6a17be64c9afc3c6e010d80e8d09062cd8ab1e1144f9c5dae8291e9cc0 Jul 02 20:24:06.617985 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:06.617945 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6" event={"ID":"c28875e4-241d-4f91-a517-83d78a5c4dd6","Type":"ContainerStarted","Data":"7ca3da6a17be64c9afc3c6e010d80e8d09062cd8ab1e1144f9c5dae8291e9cc0"} Jul 02 20:24:08.628582 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:08.628553 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6" event={"ID":"c28875e4-241d-4f91-a517-83d78a5c4dd6","Type":"ContainerStarted","Data":"760e5c053fc946e7af14a55135219801d6ef48d1ec2b2818baccafdcc943f1c1"} Jul 02 20:24:08.645946 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:08.645897 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7fvt6" podStartSLOduration=1.402304003 podStartE2EDuration="3.645882529s" totalImagesPullingTime="2.243578526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:24:08.644812996 +0000 UTC m=+557.641116228" watchObservedRunningTime="2026-07-02 20:24:08.645882529 +0000 UTC m=+557.642185761" Jul 02 20:24:09.310925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:09.310892 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:09.412165 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:09.412129 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:11.491008 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:11.490969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:23.333752 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:23.333722 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:23.431870 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:23.431841 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:23.977030 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:23.977001 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:24.225038 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.225008 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:24.328018 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.327934 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:24.392868 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.392840 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h"] Jul 02 20:24:24.399367 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.399346 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.405081 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.405059 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xncpr\"" Jul 02 20:24:24.405259 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.405234 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 02 20:24:24.405382 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.405309 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 02 20:24:24.405447 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.405389 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 02 20:24:24.405735 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.405716 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 02 20:24:24.409682 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.409619 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h"] Jul 02 20:24:24.482869 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.482840 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3929840f-933b-4207-8684-5a10e32135c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.483054 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.482908 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3929840f-933b-4207-8684-5a10e32135c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.483054 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.482938 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3929840f-933b-4207-8684-5a10e32135c0-manifests\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.483054 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.482977 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvs42\" (UniqueName: \"kubernetes.io/projected/3929840f-933b-4207-8684-5a10e32135c0-kube-api-access-cvs42\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.483054 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.483044 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3929840f-933b-4207-8684-5a10e32135c0-tmp\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.523982 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.523932 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:24.584344 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.584250 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3929840f-933b-4207-8684-5a10e32135c0-tmp\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.584344 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.584295 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3929840f-933b-4207-8684-5a10e32135c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.584344 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.584333 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3929840f-933b-4207-8684-5a10e32135c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.584613 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.584353 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3929840f-933b-4207-8684-5a10e32135c0-manifests\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.584613 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.584380 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvs42\" (UniqueName: \"kubernetes.io/projected/3929840f-933b-4207-8684-5a10e32135c0-kube-api-access-cvs42\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.584811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.584781 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3929840f-933b-4207-8684-5a10e32135c0-manifests\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.586900 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.586878 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3929840f-933b-4207-8684-5a10e32135c0-tmp\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.587114 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.587095 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3929840f-933b-4207-8684-5a10e32135c0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.587198 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.587105 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3929840f-933b-4207-8684-5a10e32135c0-webhook-cert\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.592438 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.592419 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvs42\" (UniqueName: \"kubernetes.io/projected/3929840f-933b-4207-8684-5a10e32135c0-kube-api-access-cvs42\") pod \"opendatahub-operator-controller-manager-78b478dcdf-4p54h\" (UID: \"3929840f-933b-4207-8684-5a10e32135c0\") " pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.624310 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.624285 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:24.712298 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.712268 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:24.725714 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.725692 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:24.853117 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.853035 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:24.876891 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.876869 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h"] Jul 02 20:24:24.879281 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:24:24.879253 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3929840f_933b_4207_8684_5a10e32135c0.slice/crio-856c145ba2cb20560f579b1c1e5ffdb3b7ae24a6f9f6235c407e211b6a415b65 WatchSource:0}: Error finding container 856c145ba2cb20560f579b1c1e5ffdb3b7ae24a6f9f6235c407e211b6a415b65: Status 404 returned error can't find the container with id 856c145ba2cb20560f579b1c1e5ffdb3b7ae24a6f9f6235c407e211b6a415b65 Jul 02 20:24:24.962519 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:24.962487 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:25.062205 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:25.062175 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:25.702741 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:25.702652 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" event={"ID":"3929840f-933b-4207-8684-5a10e32135c0","Type":"ContainerStarted","Data":"856c145ba2cb20560f579b1c1e5ffdb3b7ae24a6f9f6235c407e211b6a415b65"} Jul 02 20:24:25.791727 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:25.791683 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:25.891537 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:25.891501 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:28.720628 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:28.720592 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" event={"ID":"3929840f-933b-4207-8684-5a10e32135c0","Type":"ContainerStarted","Data":"c4ec650f25b19aeafe943dbfee253fba0a19c967a87d27392dbe554437acff31"} Jul 02 20:24:29.726312 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:29.726280 2565 generic.go:356] "Generic (PLEG): container finished" podID="3929840f-933b-4207-8684-5a10e32135c0" containerID="c4ec650f25b19aeafe943dbfee253fba0a19c967a87d27392dbe554437acff31" exitCode=0 Jul 02 20:24:29.726739 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:29.726371 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" event={"ID":"3929840f-933b-4207-8684-5a10e32135c0","Type":"ContainerDied","Data":"c4ec650f25b19aeafe943dbfee253fba0a19c967a87d27392dbe554437acff31"} Jul 02 20:24:30.737771 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:30.737737 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" event={"ID":"3929840f-933b-4207-8684-5a10e32135c0","Type":"ContainerStarted","Data":"b0e21fd47fbdacc839d49cfd8691654af976dee38a3403512abf4cab40646bcb"} Jul 02 20:24:30.738154 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:30.737848 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:30.764357 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:30.764312 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" podStartSLOduration=1.898885485 podStartE2EDuration="6.764298563s" totalImagesPullingTime="3.779244238s" totalInitContainerRuntime="1.08616884s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:24:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:24:30.763691022 +0000 UTC m=+579.759994256" watchObservedRunningTime="2026-07-02 20:24:30.764298563 +0000 UTC m=+579.760601798" Jul 02 20:24:36.503750 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:36.503711 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:38.070233 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:38.070189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:41.744382 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:41.744351 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-78b478dcdf-4p54h" Jul 02 20:24:43.560542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:43.560514 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:43.683843 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:43.683810 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:44.327619 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.327582 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx"] Jul 02 20:24:44.330563 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.330540 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.334382 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.334357 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 02 20:24:44.334504 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.334419 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-xnq6n\"" Jul 02 20:24:44.334504 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.334423 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 02 20:24:44.334504 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.334485 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 02 20:24:44.341539 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.341513 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60f6cb36-921e-4f3e-8598-994e01fab8c0-metrics-cert\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.341717 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.341561 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60f6cb36-921e-4f3e-8598-994e01fab8c0-manager-config\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.341717 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.341633 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pktw\" (UniqueName: \"kubernetes.io/projected/60f6cb36-921e-4f3e-8598-994e01fab8c0-kube-api-access-4pktw\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.341717 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.341672 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60f6cb36-921e-4f3e-8598-994e01fab8c0-cert\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.343422 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.343402 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx"] Jul 02 20:24:44.442569 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.442526 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60f6cb36-921e-4f3e-8598-994e01fab8c0-metrics-cert\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.442569 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.442565 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60f6cb36-921e-4f3e-8598-994e01fab8c0-manager-config\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.442794 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.442690 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pktw\" (UniqueName: \"kubernetes.io/projected/60f6cb36-921e-4f3e-8598-994e01fab8c0-kube-api-access-4pktw\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.442794 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.442735 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60f6cb36-921e-4f3e-8598-994e01fab8c0-cert\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.443193 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.443175 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/60f6cb36-921e-4f3e-8598-994e01fab8c0-manager-config\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.445308 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.445286 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/60f6cb36-921e-4f3e-8598-994e01fab8c0-metrics-cert\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.445408 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.445348 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/60f6cb36-921e-4f3e-8598-994e01fab8c0-cert\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.451613 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.451588 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pktw\" (UniqueName: \"kubernetes.io/projected/60f6cb36-921e-4f3e-8598-994e01fab8c0-kube-api-access-4pktw\") pod \"lws-controller-manager-5b47c96757-z8lzx\" (UID: \"60f6cb36-921e-4f3e-8598-994e01fab8c0\") " pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.640463 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.640366 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:44.773792 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.773765 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx"] Jul 02 20:24:44.776419 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:24:44.776382 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60f6cb36_921e_4f3e_8598_994e01fab8c0.slice/crio-495cd3ea76b6afa5d5561e4046fe078d6d255f3f8cd96701ec3581f8c892d490 WatchSource:0}: Error finding container 495cd3ea76b6afa5d5561e4046fe078d6d255f3f8cd96701ec3581f8c892d490: Status 404 returned error can't find the container with id 495cd3ea76b6afa5d5561e4046fe078d6d255f3f8cd96701ec3581f8c892d490 Jul 02 20:24:44.797802 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:44.797768 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" event={"ID":"60f6cb36-921e-4f3e-8598-994e01fab8c0","Type":"ContainerStarted","Data":"495cd3ea76b6afa5d5561e4046fe078d6d255f3f8cd96701ec3581f8c892d490"} Jul 02 20:24:46.808647 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:46.808602 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" event={"ID":"60f6cb36-921e-4f3e-8598-994e01fab8c0","Type":"ContainerStarted","Data":"92497c4de02d0102b13c2dc9b3758543f25ba7c1f694ddef4e553f20f43573a6"} Jul 02 20:24:46.808996 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:46.808741 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:46.830890 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:46.830835 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" podStartSLOduration=1.022033731 podStartE2EDuration="2.83081943s" totalImagesPullingTime="1.808785699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:24:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:24:46.828444902 +0000 UTC m=+595.824748136" watchObservedRunningTime="2026-07-02 20:24:46.83081943 +0000 UTC m=+595.827123096" Jul 02 20:24:49.284258 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.284226 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:49.456640 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.456609 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-d957c959b-4767p"] Jul 02 20:24:49.459898 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.459874 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" Jul 02 20:24:49.464092 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.464063 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 20:24:49.464280 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.464078 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 02 20:24:49.464388 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.464078 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 02 20:24:49.464388 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.464097 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 20:24:49.464388 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.464118 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4bt7t\"" Jul 02 20:24:49.478574 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.478546 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d957c959b-4767p"] Jul 02 20:24:49.487143 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.487114 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/642c49dc-9648-4b04-a6ba-b775546004ac-tmp\") pod \"kube-auth-proxy-d957c959b-4767p\" (UID: \"642c49dc-9648-4b04-a6ba-b775546004ac\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" Jul 02 20:24:49.487296 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.487162 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642c49dc-9648-4b04-a6ba-b775546004ac-tls-certs\") pod \"kube-auth-proxy-d957c959b-4767p\" (UID: \"642c49dc-9648-4b04-a6ba-b775546004ac\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" Jul 02 20:24:49.487296 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.487187 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vtqj\" (UniqueName: \"kubernetes.io/projected/642c49dc-9648-4b04-a6ba-b775546004ac-kube-api-access-2vtqj\") pod \"kube-auth-proxy-d957c959b-4767p\" (UID: \"642c49dc-9648-4b04-a6ba-b775546004ac\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" Jul 02 20:24:49.548568 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.548486 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:49.587769 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.587729 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642c49dc-9648-4b04-a6ba-b775546004ac-tls-certs\") pod \"kube-auth-proxy-d957c959b-4767p\" (UID: \"642c49dc-9648-4b04-a6ba-b775546004ac\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" Jul 02 20:24:49.587914 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.587779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vtqj\" (UniqueName: \"kubernetes.io/projected/642c49dc-9648-4b04-a6ba-b775546004ac-kube-api-access-2vtqj\") pod \"kube-auth-proxy-d957c959b-4767p\" (UID: \"642c49dc-9648-4b04-a6ba-b775546004ac\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" Jul 02 20:24:49.587914 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.587830 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/642c49dc-9648-4b04-a6ba-b775546004ac-tmp\") pod \"kube-auth-proxy-d957c959b-4767p\" (UID: \"642c49dc-9648-4b04-a6ba-b775546004ac\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" Jul 02 20:24:49.590238 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.590183 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/642c49dc-9648-4b04-a6ba-b775546004ac-tmp\") pod \"kube-auth-proxy-d957c959b-4767p\" (UID: \"642c49dc-9648-4b04-a6ba-b775546004ac\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" Jul 02 20:24:49.590353 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.590327 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/642c49dc-9648-4b04-a6ba-b775546004ac-tls-certs\") pod \"kube-auth-proxy-d957c959b-4767p\" (UID: \"642c49dc-9648-4b04-a6ba-b775546004ac\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" Jul 02 20:24:49.600479 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.600455 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vtqj\" (UniqueName: \"kubernetes.io/projected/642c49dc-9648-4b04-a6ba-b775546004ac-kube-api-access-2vtqj\") pod \"kube-auth-proxy-d957c959b-4767p\" (UID: \"642c49dc-9648-4b04-a6ba-b775546004ac\") " pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" Jul 02 20:24:49.631399 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.631374 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:49.733983 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.733951 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:49.769963 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.769927 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" Jul 02 20:24:49.831485 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.831458 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:49.912888 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.912863 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-d957c959b-4767p"] Jul 02 20:24:49.915265 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:24:49.915233 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod642c49dc_9648_4b04_a6ba_b775546004ac.slice/crio-e920be40aa0bd28ca83f0884099e8daadef6ed1199970e3a774303f1ca407190 WatchSource:0}: Error finding container e920be40aa0bd28ca83f0884099e8daadef6ed1199970e3a774303f1ca407190: Status 404 returned error can't find the container with id e920be40aa0bd28ca83f0884099e8daadef6ed1199970e3a774303f1ca407190 Jul 02 20:24:49.967300 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:49.967273 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:50.443787 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:50.443756 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:50.762260 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:50.762154 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:50.831662 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:50.831623 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" event={"ID":"642c49dc-9648-4b04-a6ba-b775546004ac","Type":"ContainerStarted","Data":"e920be40aa0bd28ca83f0884099e8daadef6ed1199970e3a774303f1ca407190"} Jul 02 20:24:50.882658 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:50.882627 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:51.445129 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:51.445102 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:24:51.445605 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:51.445201 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:24:51.922321 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:51.922288 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:52.732467 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:52.732446 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 02 20:24:52.842392 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:52.842347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" event={"ID":"642c49dc-9648-4b04-a6ba-b775546004ac","Type":"ContainerStarted","Data":"356469b2c2465f68be4ce2a648062d07495ba993c493d3e849b13f7a92f00061"} Jul 02 20:24:52.860175 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:52.860127 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-d957c959b-4767p" podStartSLOduration=1.047391693 podStartE2EDuration="3.860112531s" totalImagesPullingTime="2.812720838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:24:52.858294285 +0000 UTC m=+601.854597521" watchObservedRunningTime="2026-07-02 20:24:52.860112531 +0000 UTC m=+601.856415765" Jul 02 20:24:56.934690 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:56.934656 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:57.037489 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:57.037461 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:57.639782 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:57.639746 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:57.747697 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:57.747668 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:57.816022 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:57.815993 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5b47c96757-z8lzx" Jul 02 20:24:58.051053 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:58.051016 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:59.690777 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:59.690741 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:24:59.828810 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:24:59.828771 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:01.459639 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:01.459579 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:01.673929 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:01.673898 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:01.850154 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:01.850059 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:02.830326 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:02.830298 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:02.971025 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:02.971000 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:03.252872 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:03.252843 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:44.948805 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:44.948761 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:45.045599 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:45.045559 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:45.145322 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:45.145262 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:45.876463 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:45.876433 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:46.082922 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:46.082891 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:53.794780 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:53.794744 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:53.862586 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:53.862556 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:54.560538 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:54.560502 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:54.659881 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:54.659849 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:55.745451 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:55.745421 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:25:55.825418 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:25:55.825383 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:06.097867 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.097835 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5db4d66dfc-n4snw"] Jul 02 20:26:06.101326 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.101292 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.113333 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.113299 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db4d66dfc-n4snw"] Jul 02 20:26:06.161326 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.161299 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:06.247983 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.247950 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:06.277170 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.277139 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-console-config\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.277326 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.277176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-service-ca\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.277326 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.277204 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-trusted-ca-bundle\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.277416 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.277340 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55bbf3ab-8c87-44ca-89cd-d942b106e12c-console-serving-cert\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.277416 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.277369 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-oauth-serving-cert\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.277416 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.277393 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55bbf3ab-8c87-44ca-89cd-d942b106e12c-console-oauth-config\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.277523 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.277438 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2jkf\" (UniqueName: \"kubernetes.io/projected/55bbf3ab-8c87-44ca-89cd-d942b106e12c-kube-api-access-c2jkf\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.378593 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.378513 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-console-config\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.378593 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.378546 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-service-ca\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.378593 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.378568 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-trusted-ca-bundle\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.378859 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.378605 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55bbf3ab-8c87-44ca-89cd-d942b106e12c-console-serving-cert\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.378859 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.378734 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-oauth-serving-cert\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.378859 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.378781 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55bbf3ab-8c87-44ca-89cd-d942b106e12c-console-oauth-config\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.378859 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.378856 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2jkf\" (UniqueName: \"kubernetes.io/projected/55bbf3ab-8c87-44ca-89cd-d942b106e12c-kube-api-access-c2jkf\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.379383 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.379358 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-console-config\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.379532 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.379509 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-oauth-serving-cert\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.379607 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.379518 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-trusted-ca-bundle\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.379607 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.379575 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/55bbf3ab-8c87-44ca-89cd-d942b106e12c-service-ca\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.381180 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.381154 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/55bbf3ab-8c87-44ca-89cd-d942b106e12c-console-serving-cert\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.381297 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.381275 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/55bbf3ab-8c87-44ca-89cd-d942b106e12c-console-oauth-config\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.387006 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.386986 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2jkf\" (UniqueName: \"kubernetes.io/projected/55bbf3ab-8c87-44ca-89cd-d942b106e12c-kube-api-access-c2jkf\") pod \"console-5db4d66dfc-n4snw\" (UID: \"55bbf3ab-8c87-44ca-89cd-d942b106e12c\") " pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.411541 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.411521 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:06.546675 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:06.546649 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5db4d66dfc-n4snw"] Jul 02 20:26:06.548433 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:26:06.548399 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bbf3ab_8c87_44ca_89cd_d942b106e12c.slice/crio-1285b44aaddc490dd885bfc9664af5863930e3111e1d2e06ceb789784c62e096 WatchSource:0}: Error finding container 1285b44aaddc490dd885bfc9664af5863930e3111e1d2e06ceb789784c62e096: Status 404 returned error can't find the container with id 1285b44aaddc490dd885bfc9664af5863930e3111e1d2e06ceb789784c62e096 Jul 02 20:26:07.187244 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:07.187186 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db4d66dfc-n4snw" event={"ID":"55bbf3ab-8c87-44ca-89cd-d942b106e12c","Type":"ContainerStarted","Data":"57e635e1f90be241d10c55f83d361a23b878199502c17839d92a8a43d17cc92a"} Jul 02 20:26:07.187636 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:07.187268 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5db4d66dfc-n4snw" event={"ID":"55bbf3ab-8c87-44ca-89cd-d942b106e12c","Type":"ContainerStarted","Data":"1285b44aaddc490dd885bfc9664af5863930e3111e1d2e06ceb789784c62e096"} Jul 02 20:26:07.214996 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:07.214931 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5db4d66dfc-n4snw" podStartSLOduration=1.214912282 podStartE2EDuration="1.214912282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:26:07.21436642 +0000 UTC m=+676.210669654" watchObservedRunningTime="2026-07-02 20:26:07.214912282 +0000 UTC m=+676.211215516" Jul 02 20:26:14.120237 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:14.120176 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:16.412687 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:16.412654 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:16.413256 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:16.413004 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:16.417584 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:16.417553 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:17.234939 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:17.234909 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5db4d66dfc-n4snw" Jul 02 20:26:17.345851 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:17.345814 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7c85466b-56gk7"] Jul 02 20:26:17.409392 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:17.409344 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:17.556536 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:17.556456 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:17.706020 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:17.705987 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:19.166320 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:19.166289 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:19.518531 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:19.518436 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:20.165110 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:20.165073 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:22.154001 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:22.153964 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:24.143182 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:24.143150 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:24.430736 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:24.430703 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:27.166596 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:27.166565 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:28.870169 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:28.870137 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:29.118823 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.118773 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:29.613087 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.613051 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp"] Jul 02 20:26:29.616823 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.616799 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:29.620340 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.620311 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 20:26:29.620554 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.620526 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 02 20:26:29.620662 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.620560 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 02 20:26:29.620662 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.620574 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hzkvd\"" Jul 02 20:26:29.620662 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.620573 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 20:26:29.636427 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.636401 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp"] Jul 02 20:26:29.674729 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.674699 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-d9hxp\" (UID: \"d6b2d93d-1f86-4f70-8977-03c7017bf5c9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:29.674882 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.674737 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gknnf\" (UniqueName: \"kubernetes.io/projected/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-kube-api-access-gknnf\") pod \"kuadrant-console-plugin-66b5774985-d9hxp\" (UID: \"d6b2d93d-1f86-4f70-8977-03c7017bf5c9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:29.674882 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.674770 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-d9hxp\" (UID: \"d6b2d93d-1f86-4f70-8977-03c7017bf5c9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:29.740429 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.740405 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:29.775750 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.775721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-d9hxp\" (UID: \"d6b2d93d-1f86-4f70-8977-03c7017bf5c9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:29.775869 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.775764 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gknnf\" (UniqueName: \"kubernetes.io/projected/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-kube-api-access-gknnf\") pod \"kuadrant-console-plugin-66b5774985-d9hxp\" (UID: \"d6b2d93d-1f86-4f70-8977-03c7017bf5c9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:29.775869 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.775800 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-d9hxp\" (UID: \"d6b2d93d-1f86-4f70-8977-03c7017bf5c9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:29.776011 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:26:29.775863 2565 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 02 20:26:29.776011 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:26:29.775946 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-plugin-serving-cert podName:d6b2d93d-1f86-4f70-8977-03c7017bf5c9 nodeName:}" failed. No retries permitted until 2026-07-02 20:26:30.275924868 +0000 UTC m=+699.272228078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-d9hxp" (UID: "d6b2d93d-1f86-4f70-8977-03c7017bf5c9") : secret "plugin-serving-cert" not found Jul 02 20:26:29.776448 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.776433 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-d9hxp\" (UID: \"d6b2d93d-1f86-4f70-8977-03c7017bf5c9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:29.809935 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.809915 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gknnf\" (UniqueName: \"kubernetes.io/projected/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-kube-api-access-gknnf\") pod \"kuadrant-console-plugin-66b5774985-d9hxp\" (UID: \"d6b2d93d-1f86-4f70-8977-03c7017bf5c9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:29.927865 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:29.927835 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:30.280497 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:30.280416 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-d9hxp\" (UID: \"d6b2d93d-1f86-4f70-8977-03c7017bf5c9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:30.282997 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:30.282974 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/d6b2d93d-1f86-4f70-8977-03c7017bf5c9-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-d9hxp\" (UID: \"d6b2d93d-1f86-4f70-8977-03c7017bf5c9\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:30.526634 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:30.526594 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" Jul 02 20:26:30.676541 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:30.676511 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp"] Jul 02 20:26:30.679635 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:26:30.679610 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6b2d93d_1f86_4f70_8977_03c7017bf5c9.slice/crio-ef2efcde89ef8ad961a141bb9601f5db368b724152c38c30e26a14d58a9bca84 WatchSource:0}: Error finding container ef2efcde89ef8ad961a141bb9601f5db368b724152c38c30e26a14d58a9bca84: Status 404 returned error can't find the container with id ef2efcde89ef8ad961a141bb9601f5db368b724152c38c30e26a14d58a9bca84 Jul 02 20:26:30.681295 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:30.681278 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:26:31.294293 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:31.294259 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" event={"ID":"d6b2d93d-1f86-4f70-8977-03c7017bf5c9","Type":"ContainerStarted","Data":"ef2efcde89ef8ad961a141bb9601f5db368b724152c38c30e26a14d58a9bca84"} Jul 02 20:26:37.645380 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:37.645343 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:37.846058 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:37.846005 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:37.994581 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:37.994550 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:40.757585 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:40.757535 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:40.901040 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:40.901007 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:42.367406 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:42.367301 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6c7c85466b-56gk7" podUID="d8130ea0-86fb-4b36-b746-cc0129a91f1a" containerName="console" containerID="cri-o://7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859" gracePeriod=15 Jul 02 20:26:44.618293 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:44.618266 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:44.733167 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:44.733132 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:45.461380 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:45.461341 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:45.560606 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:45.560570 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:46.343689 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:46.343649 2565 patch_prober.go:28] interesting pod/console-6c7c85466b-56gk7 container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.24:8443/health\": dial tcp 10.133.0.24:8443: connect: connection refused" start-of-body= Jul 02 20:26:46.344131 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:46.343720 2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6c7c85466b-56gk7" podUID="d8130ea0-86fb-4b36-b746-cc0129a91f1a" containerName="console" probeResult="failure" output="Get \"https://10.133.0.24:8443/health\": dial tcp 10.133.0.24:8443: connect: connection refused" Jul 02 20:26:48.552161 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:48.552109 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:48.672898 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:48.672817 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:53.743422 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.743395 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:26:53.913184 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913155 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-serving-cert\") pod \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " Jul 02 20:26:53.913348 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913237 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-config\") pod \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " Jul 02 20:26:53.913348 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913332 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-trusted-ca-bundle\") pod \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " Jul 02 20:26:53.913443 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913380 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-service-ca\") pod \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " Jul 02 20:26:53.913443 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913416 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-oauth-serving-cert\") pod \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " Jul 02 20:26:53.913550 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913461 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z68b8\" (UniqueName: \"kubernetes.io/projected/d8130ea0-86fb-4b36-b746-cc0129a91f1a-kube-api-access-z68b8\") pod \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " Jul 02 20:26:53.913550 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913497 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-oauth-config\") pod \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\" (UID: \"d8130ea0-86fb-4b36-b746-cc0129a91f1a\") " Jul 02 20:26:53.913693 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913672 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-config" (OuterVolumeSpecName: "console-config") pod "d8130ea0-86fb-4b36-b746-cc0129a91f1a" (UID: "d8130ea0-86fb-4b36-b746-cc0129a91f1a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:26:53.913818 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913786 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d8130ea0-86fb-4b36-b746-cc0129a91f1a" (UID: "d8130ea0-86fb-4b36-b746-cc0129a91f1a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:26:53.913818 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913797 2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-config\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:26:53.913943 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913855 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-service-ca" (OuterVolumeSpecName: "service-ca") pod "d8130ea0-86fb-4b36-b746-cc0129a91f1a" (UID: "d8130ea0-86fb-4b36-b746-cc0129a91f1a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:26:53.913943 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.913916 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d8130ea0-86fb-4b36-b746-cc0129a91f1a" (UID: "d8130ea0-86fb-4b36-b746-cc0129a91f1a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:26:53.915442 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.915415 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d8130ea0-86fb-4b36-b746-cc0129a91f1a" (UID: "d8130ea0-86fb-4b36-b746-cc0129a91f1a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:26:53.915654 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.915626 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8130ea0-86fb-4b36-b746-cc0129a91f1a-kube-api-access-z68b8" (OuterVolumeSpecName: "kube-api-access-z68b8") pod "d8130ea0-86fb-4b36-b746-cc0129a91f1a" (UID: "d8130ea0-86fb-4b36-b746-cc0129a91f1a"). InnerVolumeSpecName "kube-api-access-z68b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:26:53.915797 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:53.915779 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d8130ea0-86fb-4b36-b746-cc0129a91f1a" (UID: "d8130ea0-86fb-4b36-b746-cc0129a91f1a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:26:54.014685 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.014653 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68b8\" (UniqueName: \"kubernetes.io/projected/d8130ea0-86fb-4b36-b746-cc0129a91f1a-kube-api-access-z68b8\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:26:54.014837 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.014699 2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-oauth-config\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:26:54.014837 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.014710 2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d8130ea0-86fb-4b36-b746-cc0129a91f1a-console-serving-cert\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:26:54.014837 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.014719 2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-trusted-ca-bundle\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:26:54.014837 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.014727 2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-service-ca\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:26:54.014837 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.014735 2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d8130ea0-86fb-4b36-b746-cc0129a91f1a-oauth-serving-cert\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:26:54.426285 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.426253 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" event={"ID":"d6b2d93d-1f86-4f70-8977-03c7017bf5c9","Type":"ContainerStarted","Data":"86114b6d254045d9cca0a0ae86a321abb12af23b078ee36e032874620980ac16"} Jul 02 20:26:54.427572 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.427549 2565 generic.go:356] "Generic (PLEG): container finished" podID="d8130ea0-86fb-4b36-b746-cc0129a91f1a" containerID="7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859" exitCode=0 Jul 02 20:26:54.427674 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.427585 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c7c85466b-56gk7" Jul 02 20:26:54.427674 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.427593 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7c85466b-56gk7" event={"ID":"d8130ea0-86fb-4b36-b746-cc0129a91f1a","Type":"ContainerDied","Data":"7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859"} Jul 02 20:26:54.427674 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.427624 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c7c85466b-56gk7" event={"ID":"d8130ea0-86fb-4b36-b746-cc0129a91f1a","Type":"ContainerDied","Data":"cdc65f68186c5aec851e89461a39becca9a2bca8416926eb2872b2dc30e479f0"} Jul 02 20:26:54.427674 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.427644 2565 scope.go:117] "RemoveContainer" containerID="7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859" Jul 02 20:26:54.436433 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.436417 2565 scope.go:117] "RemoveContainer" containerID="7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859" Jul 02 20:26:54.436710 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:26:54.436680 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859\": container with ID starting with 7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859 not found: ID does not exist" containerID="7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859" Jul 02 20:26:54.436776 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.436723 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859"} err="failed to get container status \"7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859\": rpc error: code = NotFound desc = could not find container \"7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859\": container with ID starting with 7d823742c114aace56cfc7aeb6e152b206b4df6825b3b64531983bfb299b2859 not found: ID does not exist" Jul 02 20:26:54.445440 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.445398 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-d9hxp" podStartSLOduration=2.510847126 podStartE2EDuration="25.44538505s" totalImagesPullingTime="22.934537924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:26:54.444104437 +0000 UTC m=+723.440407683" watchObservedRunningTime="2026-07-02 20:26:54.44538505 +0000 UTC m=+723.441688283" Jul 02 20:26:54.466012 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.465979 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c7c85466b-56gk7"] Jul 02 20:26:54.469694 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:54.469669 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c7c85466b-56gk7"] Jul 02 20:26:55.571498 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:55.571470 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8130ea0-86fb-4b36-b746-cc0129a91f1a" path="/var/lib/kubelet/pods/d8130ea0-86fb-4b36-b746-cc0129a91f1a/volumes" Jul 02 20:26:57.353414 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:57.353382 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:57.452436 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:57.452401 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:59.771031 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:59.770998 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:26:59.857107 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:26:59.857082 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:04.551285 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:04.551249 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:04.664024 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:04.663989 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:08.739115 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:08.739079 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:08.838089 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:08.838057 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:12.735940 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:12.735891 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:12.870887 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:12.870854 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:17.438191 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.438156 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:17.462201 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.462162 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mw9cn"] Jul 02 20:27:17.462607 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.462589 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8130ea0-86fb-4b36-b746-cc0129a91f1a" containerName="console" Jul 02 20:27:17.462693 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.462611 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8130ea0-86fb-4b36-b746-cc0129a91f1a" containerName="console" Jul 02 20:27:17.462693 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.462686 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8130ea0-86fb-4b36-b746-cc0129a91f1a" containerName="console" Jul 02 20:27:17.464639 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.464619 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:17.468404 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.468387 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 02 20:27:17.483272 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.483242 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mw9cn"] Jul 02 20:27:17.510402 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.510372 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/296bdba5-3271-4194-9d06-2f7ff688ccd2-config-file\") pod \"limitador-limitador-64c94fd64f-mw9cn\" (UID: \"296bdba5-3271-4194-9d06-2f7ff688ccd2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:17.510566 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.510432 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmf68\" (UniqueName: \"kubernetes.io/projected/296bdba5-3271-4194-9d06-2f7ff688ccd2-kube-api-access-qmf68\") pod \"limitador-limitador-64c94fd64f-mw9cn\" (UID: \"296bdba5-3271-4194-9d06-2f7ff688ccd2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:17.542176 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.542148 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mw9cn"] Jul 02 20:27:17.565495 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.565459 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:17.611591 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.611558 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/296bdba5-3271-4194-9d06-2f7ff688ccd2-config-file\") pod \"limitador-limitador-64c94fd64f-mw9cn\" (UID: \"296bdba5-3271-4194-9d06-2f7ff688ccd2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:17.611762 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.611631 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmf68\" (UniqueName: \"kubernetes.io/projected/296bdba5-3271-4194-9d06-2f7ff688ccd2-kube-api-access-qmf68\") pod \"limitador-limitador-64c94fd64f-mw9cn\" (UID: \"296bdba5-3271-4194-9d06-2f7ff688ccd2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:17.612186 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.612168 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/296bdba5-3271-4194-9d06-2f7ff688ccd2-config-file\") pod \"limitador-limitador-64c94fd64f-mw9cn\" (UID: \"296bdba5-3271-4194-9d06-2f7ff688ccd2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:17.621929 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.621905 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmf68\" (UniqueName: \"kubernetes.io/projected/296bdba5-3271-4194-9d06-2f7ff688ccd2-kube-api-access-qmf68\") pod \"limitador-limitador-64c94fd64f-mw9cn\" (UID: \"296bdba5-3271-4194-9d06-2f7ff688ccd2\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:17.671158 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.671126 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:17.768361 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.768261 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:17.774604 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.774578 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:17.864650 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.864602 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:17.912863 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:17.912843 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mw9cn"] Jul 02 20:27:17.915183 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:27:17.915158 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296bdba5_3271_4194_9d06_2f7ff688ccd2.slice/crio-1fe12e0dbdaa7eaf23f05f6c00e9e5147901e20bf839549f1e8f05f551e518dc WatchSource:0}: Error finding container 1fe12e0dbdaa7eaf23f05f6c00e9e5147901e20bf839549f1e8f05f551e518dc: Status 404 returned error can't find the container with id 1fe12e0dbdaa7eaf23f05f6c00e9e5147901e20bf839549f1e8f05f551e518dc Jul 02 20:27:18.390068 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.390039 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-8fmtg"] Jul 02 20:27:18.401667 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.401642 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8fmtg" Jul 02 20:27:18.405795 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.405705 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ttq9h\"" Jul 02 20:27:18.412012 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.411993 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8fmtg"] Jul 02 20:27:18.473707 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.473673 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:18.520549 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.520524 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxvh7\" (UniqueName: \"kubernetes.io/projected/ce6ea24a-4752-407a-b83b-f9e473be46f8-kube-api-access-hxvh7\") pod \"authorino-58cd955675-8fmtg\" (UID: \"ce6ea24a-4752-407a-b83b-f9e473be46f8\") " pod="kuadrant-system/authorino-58cd955675-8fmtg" Jul 02 20:27:18.535370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.535347 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" event={"ID":"296bdba5-3271-4194-9d06-2f7ff688ccd2","Type":"ContainerStarted","Data":"1fe12e0dbdaa7eaf23f05f6c00e9e5147901e20bf839549f1e8f05f551e518dc"} Jul 02 20:27:18.564253 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.564232 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:18.621520 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.621488 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxvh7\" (UniqueName: \"kubernetes.io/projected/ce6ea24a-4752-407a-b83b-f9e473be46f8-kube-api-access-hxvh7\") pod \"authorino-58cd955675-8fmtg\" (UID: \"ce6ea24a-4752-407a-b83b-f9e473be46f8\") " pod="kuadrant-system/authorino-58cd955675-8fmtg" Jul 02 20:27:18.635775 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.635747 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxvh7\" (UniqueName: \"kubernetes.io/projected/ce6ea24a-4752-407a-b83b-f9e473be46f8-kube-api-access-hxvh7\") pod \"authorino-58cd955675-8fmtg\" (UID: \"ce6ea24a-4752-407a-b83b-f9e473be46f8\") " pod="kuadrant-system/authorino-58cd955675-8fmtg" Jul 02 20:27:18.712763 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.712726 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8fmtg" Jul 02 20:27:18.909029 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:18.908956 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8fmtg"] Jul 02 20:27:18.914329 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:27:18.914297 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce6ea24a_4752_407a_b83b_f9e473be46f8.slice/crio-7dc0b531c9d509c9e8476e79357d4c3608cc5af6822f19e002aa201b5d0ad43e WatchSource:0}: Error finding container 7dc0b531c9d509c9e8476e79357d4c3608cc5af6822f19e002aa201b5d0ad43e: Status 404 returned error can't find the container with id 7dc0b531c9d509c9e8476e79357d4c3608cc5af6822f19e002aa201b5d0ad43e Jul 02 20:27:19.542221 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:19.542180 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8fmtg" event={"ID":"ce6ea24a-4752-407a-b83b-f9e473be46f8","Type":"ContainerStarted","Data":"7dc0b531c9d509c9e8476e79357d4c3608cc5af6822f19e002aa201b5d0ad43e"} Jul 02 20:27:20.205280 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:20.205248 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:20.305970 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:20.305917 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:21.557012 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:21.556976 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" event={"ID":"296bdba5-3271-4194-9d06-2f7ff688ccd2","Type":"ContainerStarted","Data":"bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a"} Jul 02 20:27:21.557469 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:21.557048 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:21.579480 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:21.579424 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" podStartSLOduration=1.7564418339999999 podStartE2EDuration="4.57940521s" totalImagesPullingTime="2.822963376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:27:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:27:21.577276628 +0000 UTC m=+750.573579862" watchObservedRunningTime="2026-07-02 20:27:21.57940521 +0000 UTC m=+750.575708443" Jul 02 20:27:23.571301 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:23.571224 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8fmtg" event={"ID":"ce6ea24a-4752-407a-b83b-f9e473be46f8","Type":"ContainerStarted","Data":"1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a"} Jul 02 20:27:23.589808 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:23.585528 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-8fmtg" podStartSLOduration=1.300586914 podStartE2EDuration="5.585508643s" totalImagesPullingTime="4.284921729s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:27:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:27:23.583513199 +0000 UTC m=+752.579816432" watchObservedRunningTime="2026-07-02 20:27:23.585508643 +0000 UTC m=+752.581811877" Jul 02 20:27:24.249614 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:24.249581 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8fmtg"] Jul 02 20:27:24.979204 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:24.979172 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:25.085458 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:25.085428 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:25.576093 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:25.576058 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-8fmtg" podUID="ce6ea24a-4752-407a-b83b-f9e473be46f8" containerName="authorino" containerID="cri-o://1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a" gracePeriod=30 Jul 02 20:27:25.823851 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:25.823822 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8fmtg" Jul 02 20:27:25.889459 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:25.889381 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hxvh7\" (UniqueName: \"kubernetes.io/projected/ce6ea24a-4752-407a-b83b-f9e473be46f8-kube-api-access-hxvh7\") pod \"ce6ea24a-4752-407a-b83b-f9e473be46f8\" (UID: \"ce6ea24a-4752-407a-b83b-f9e473be46f8\") " Jul 02 20:27:25.891589 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:25.891562 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce6ea24a-4752-407a-b83b-f9e473be46f8-kube-api-access-hxvh7" (OuterVolumeSpecName: "kube-api-access-hxvh7") pod "ce6ea24a-4752-407a-b83b-f9e473be46f8" (UID: "ce6ea24a-4752-407a-b83b-f9e473be46f8"). InnerVolumeSpecName "kube-api-access-hxvh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:27:25.990902 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:25.990867 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxvh7\" (UniqueName: \"kubernetes.io/projected/ce6ea24a-4752-407a-b83b-f9e473be46f8-kube-api-access-hxvh7\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:27:26.581974 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:26.581942 2565 generic.go:356] "Generic (PLEG): container finished" podID="ce6ea24a-4752-407a-b83b-f9e473be46f8" containerID="1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a" exitCode=0 Jul 02 20:27:26.581974 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:26.581984 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8fmtg" Jul 02 20:27:26.582238 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:26.582020 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8fmtg" event={"ID":"ce6ea24a-4752-407a-b83b-f9e473be46f8","Type":"ContainerDied","Data":"1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a"} Jul 02 20:27:26.582238 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:26.582063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8fmtg" event={"ID":"ce6ea24a-4752-407a-b83b-f9e473be46f8","Type":"ContainerDied","Data":"7dc0b531c9d509c9e8476e79357d4c3608cc5af6822f19e002aa201b5d0ad43e"} Jul 02 20:27:26.582238 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:26.582085 2565 scope.go:117] "RemoveContainer" containerID="1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a" Jul 02 20:27:26.592809 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:26.592790 2565 scope.go:117] "RemoveContainer" containerID="1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a" Jul 02 20:27:26.593113 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:27:26.593094 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a\": container with ID starting with 1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a not found: ID does not exist" containerID="1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a" Jul 02 20:27:26.593165 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:26.593123 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a"} err="failed to get container status \"1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a\": rpc error: code = NotFound desc = could not find container \"1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a\": container with ID starting with 1bde2e1b808e61b7686132b229d2fa85681be5eef4e3ccb08e0af62ff8e7dc9a not found: ID does not exist" Jul 02 20:27:26.607830 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:26.607795 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8fmtg"] Jul 02 20:27:26.614408 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:26.614387 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-8fmtg"] Jul 02 20:27:27.571448 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:27.571417 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce6ea24a-4752-407a-b83b-f9e473be46f8" path="/var/lib/kubelet/pods/ce6ea24a-4752-407a-b83b-f9e473be46f8/volumes" Jul 02 20:27:32.244676 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:32.244639 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mw9cn"] Jul 02 20:27:32.245248 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:32.244832 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" podUID="296bdba5-3271-4194-9d06-2f7ff688ccd2" containerName="limitador" containerID="cri-o://bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a" gracePeriod=30 Jul 02 20:27:32.245608 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:32.245540 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:32.794253 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:32.794225 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:32.954812 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:32.954781 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmf68\" (UniqueName: \"kubernetes.io/projected/296bdba5-3271-4194-9d06-2f7ff688ccd2-kube-api-access-qmf68\") pod \"296bdba5-3271-4194-9d06-2f7ff688ccd2\" (UID: \"296bdba5-3271-4194-9d06-2f7ff688ccd2\") " Jul 02 20:27:32.955007 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:32.954827 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/296bdba5-3271-4194-9d06-2f7ff688ccd2-config-file\") pod \"296bdba5-3271-4194-9d06-2f7ff688ccd2\" (UID: \"296bdba5-3271-4194-9d06-2f7ff688ccd2\") " Jul 02 20:27:32.955279 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:32.955204 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/296bdba5-3271-4194-9d06-2f7ff688ccd2-config-file" (OuterVolumeSpecName: "config-file") pod "296bdba5-3271-4194-9d06-2f7ff688ccd2" (UID: "296bdba5-3271-4194-9d06-2f7ff688ccd2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:27:32.957081 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:32.957058 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/296bdba5-3271-4194-9d06-2f7ff688ccd2-kube-api-access-qmf68" (OuterVolumeSpecName: "kube-api-access-qmf68") pod "296bdba5-3271-4194-9d06-2f7ff688ccd2" (UID: "296bdba5-3271-4194-9d06-2f7ff688ccd2"). InnerVolumeSpecName "kube-api-access-qmf68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:27:33.056261 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.056197 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmf68\" (UniqueName: \"kubernetes.io/projected/296bdba5-3271-4194-9d06-2f7ff688ccd2-kube-api-access-qmf68\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:27:33.056261 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.056254 2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/296bdba5-3271-4194-9d06-2f7ff688ccd2-config-file\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:27:33.616408 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.616374 2565 generic.go:356] "Generic (PLEG): container finished" podID="296bdba5-3271-4194-9d06-2f7ff688ccd2" containerID="bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a" exitCode=0 Jul 02 20:27:33.616850 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.616419 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" Jul 02 20:27:33.616850 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.616466 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" event={"ID":"296bdba5-3271-4194-9d06-2f7ff688ccd2","Type":"ContainerDied","Data":"bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a"} Jul 02 20:27:33.616850 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.616526 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mw9cn" event={"ID":"296bdba5-3271-4194-9d06-2f7ff688ccd2","Type":"ContainerDied","Data":"1fe12e0dbdaa7eaf23f05f6c00e9e5147901e20bf839549f1e8f05f551e518dc"} Jul 02 20:27:33.616850 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.616558 2565 scope.go:117] "RemoveContainer" containerID="bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a" Jul 02 20:27:33.625134 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.625116 2565 scope.go:117] "RemoveContainer" containerID="bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a" Jul 02 20:27:33.625418 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:27:33.625401 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a\": container with ID starting with bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a not found: ID does not exist" containerID="bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a" Jul 02 20:27:33.625482 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.625427 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a"} err="failed to get container status \"bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a\": rpc error: code = NotFound desc = could not find container \"bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a\": container with ID starting with bfe93c8e604996e5f0a1ae46cb229e71c2e8d759f9096ae7d9fb8fd5f0ed2b3a not found: ID does not exist" Jul 02 20:27:33.637280 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.637254 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mw9cn"] Jul 02 20:27:33.642086 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.642062 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mw9cn"] Jul 02 20:27:33.867602 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.867519 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-h9t5q"] Jul 02 20:27:33.867901 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.867887 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="296bdba5-3271-4194-9d06-2f7ff688ccd2" containerName="limitador" Jul 02 20:27:33.867958 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.867903 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="296bdba5-3271-4194-9d06-2f7ff688ccd2" containerName="limitador" Jul 02 20:27:33.867958 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.867913 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce6ea24a-4752-407a-b83b-f9e473be46f8" containerName="authorino" Jul 02 20:27:33.867958 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.867920 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce6ea24a-4752-407a-b83b-f9e473be46f8" containerName="authorino" Jul 02 20:27:33.868052 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.867993 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="296bdba5-3271-4194-9d06-2f7ff688ccd2" containerName="limitador" Jul 02 20:27:33.868052 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.868007 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce6ea24a-4752-407a-b83b-f9e473be46f8" containerName="authorino" Jul 02 20:27:33.872283 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.872267 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-h9t5q" Jul 02 20:27:33.875465 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.875443 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 02 20:27:33.875573 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.875513 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-jnnr9\"" Jul 02 20:27:33.880144 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.880124 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-h9t5q"] Jul 02 20:27:33.906111 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.906073 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:33.963745 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.963709 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/98c73987-16d5-4082-bff8-ebfbf34c0406-data\") pod \"postgres-69fb4594cf-h9t5q\" (UID: \"98c73987-16d5-4082-bff8-ebfbf34c0406\") " pod="opendatahub/postgres-69fb4594cf-h9t5q" Jul 02 20:27:33.963907 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:33.963796 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9hr\" (UniqueName: \"kubernetes.io/projected/98c73987-16d5-4082-bff8-ebfbf34c0406-kube-api-access-6v9hr\") pod \"postgres-69fb4594cf-h9t5q\" (UID: \"98c73987-16d5-4082-bff8-ebfbf34c0406\") " pod="opendatahub/postgres-69fb4594cf-h9t5q" Jul 02 20:27:34.064328 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:34.064296 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9hr\" (UniqueName: \"kubernetes.io/projected/98c73987-16d5-4082-bff8-ebfbf34c0406-kube-api-access-6v9hr\") pod \"postgres-69fb4594cf-h9t5q\" (UID: \"98c73987-16d5-4082-bff8-ebfbf34c0406\") " pod="opendatahub/postgres-69fb4594cf-h9t5q" Jul 02 20:27:34.064503 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:34.064347 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/98c73987-16d5-4082-bff8-ebfbf34c0406-data\") pod \"postgres-69fb4594cf-h9t5q\" (UID: \"98c73987-16d5-4082-bff8-ebfbf34c0406\") " pod="opendatahub/postgres-69fb4594cf-h9t5q" Jul 02 20:27:34.064679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:34.064661 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/98c73987-16d5-4082-bff8-ebfbf34c0406-data\") pod \"postgres-69fb4594cf-h9t5q\" (UID: \"98c73987-16d5-4082-bff8-ebfbf34c0406\") " pod="opendatahub/postgres-69fb4594cf-h9t5q" Jul 02 20:27:34.074675 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:34.074652 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9hr\" (UniqueName: \"kubernetes.io/projected/98c73987-16d5-4082-bff8-ebfbf34c0406-kube-api-access-6v9hr\") pod \"postgres-69fb4594cf-h9t5q\" (UID: \"98c73987-16d5-4082-bff8-ebfbf34c0406\") " pod="opendatahub/postgres-69fb4594cf-h9t5q" Jul 02 20:27:34.183954 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:34.183918 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-h9t5q" Jul 02 20:27:34.333256 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:34.333231 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-h9t5q"] Jul 02 20:27:34.334562 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:27:34.334534 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98c73987_16d5_4082_bff8_ebfbf34c0406.slice/crio-61a6367941cc01b72d5124d078fb45f02b8e0c2a0487e990e7f8e7496a02e03d WatchSource:0}: Error finding container 61a6367941cc01b72d5124d078fb45f02b8e0c2a0487e990e7f8e7496a02e03d: Status 404 returned error can't find the container with id 61a6367941cc01b72d5124d078fb45f02b8e0c2a0487e990e7f8e7496a02e03d Jul 02 20:27:34.622841 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:34.622751 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-h9t5q" event={"ID":"98c73987-16d5-4082-bff8-ebfbf34c0406","Type":"ContainerStarted","Data":"61a6367941cc01b72d5124d078fb45f02b8e0c2a0487e990e7f8e7496a02e03d"} Jul 02 20:27:35.570601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:35.570558 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="296bdba5-3271-4194-9d06-2f7ff688ccd2" path="/var/lib/kubelet/pods/296bdba5-3271-4194-9d06-2f7ff688ccd2/volumes" Jul 02 20:27:40.656008 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:40.655975 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-h9t5q" event={"ID":"98c73987-16d5-4082-bff8-ebfbf34c0406","Type":"ContainerStarted","Data":"13074338319a1702264ae77dd119b420173d743ad975ce387f9f4ad1d144293e"} Jul 02 20:27:40.656456 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:40.656073 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-h9t5q" Jul 02 20:27:40.674521 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:40.674473 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-h9t5q" podStartSLOduration=2.161100677 podStartE2EDuration="7.674457312s" totalImagesPullingTime="5.513356635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:27:40.672565991 +0000 UTC m=+769.668869225" watchObservedRunningTime="2026-07-02 20:27:40.674457312 +0000 UTC m=+769.670760544" Jul 02 20:27:46.690082 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:46.690053 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-h9t5q" Jul 02 20:27:47.041615 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.041530 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:47.278122 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.278089 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lvgfs"] Jul 02 20:27:47.284348 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.284327 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lvgfs" Jul 02 20:27:47.293954 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.293902 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ttq9h\"" Jul 02 20:27:47.302105 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.302086 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lvgfs"] Jul 02 20:27:47.369872 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.369842 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:47.386592 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.386552 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbplt\" (UniqueName: \"kubernetes.io/projected/bc3f02f3-4e88-4077-b84f-1658aab43f10-kube-api-access-xbplt\") pod \"authorino-5d4b67c446-lvgfs\" (UID: \"bc3f02f3-4e88-4077-b84f-1658aab43f10\") " pod="kuadrant-system/authorino-5d4b67c446-lvgfs" Jul 02 20:27:47.487116 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.487075 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbplt\" (UniqueName: \"kubernetes.io/projected/bc3f02f3-4e88-4077-b84f-1658aab43f10-kube-api-access-xbplt\") pod \"authorino-5d4b67c446-lvgfs\" (UID: \"bc3f02f3-4e88-4077-b84f-1658aab43f10\") " pod="kuadrant-system/authorino-5d4b67c446-lvgfs" Jul 02 20:27:47.497344 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.497314 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbplt\" (UniqueName: \"kubernetes.io/projected/bc3f02f3-4e88-4077-b84f-1658aab43f10-kube-api-access-xbplt\") pod \"authorino-5d4b67c446-lvgfs\" (UID: \"bc3f02f3-4e88-4077-b84f-1658aab43f10\") " pod="kuadrant-system/authorino-5d4b67c446-lvgfs" Jul 02 20:27:47.534908 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.534877 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lvgfs"] Jul 02 20:27:47.535203 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.535189 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lvgfs" Jul 02 20:27:47.577744 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.577713 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5699f7bdf6-kqh58"] Jul 02 20:27:47.582054 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.582030 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5699f7bdf6-kqh58" Jul 02 20:27:47.601056 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.601025 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5699f7bdf6-kqh58"] Jul 02 20:27:47.679464 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.679433 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lvgfs"] Jul 02 20:27:47.682727 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.682705 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5699f7bdf6-kqh58"] Jul 02 20:27:47.683049 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:27:47.683017 2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-djlfj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5699f7bdf6-kqh58" podUID="03b5a35f-f5df-47fc-a522-55e87143a70a" Jul 02 20:27:47.683519 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:27:47.683497 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc3f02f3_4e88_4077_b84f_1658aab43f10.slice/crio-d7c283e0b00376cffa1004a5888afd15cc5fd23850d7b8a4f5f663a7612040cb WatchSource:0}: Error finding container d7c283e0b00376cffa1004a5888afd15cc5fd23850d7b8a4f5f663a7612040cb: Status 404 returned error can't find the container with id d7c283e0b00376cffa1004a5888afd15cc5fd23850d7b8a4f5f663a7612040cb Jul 02 20:27:47.687527 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.687506 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5699f7bdf6-kqh58" Jul 02 20:27:47.688311 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.688290 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djlfj\" (UniqueName: \"kubernetes.io/projected/03b5a35f-f5df-47fc-a522-55e87143a70a-kube-api-access-djlfj\") pod \"authorino-5699f7bdf6-kqh58\" (UID: \"03b5a35f-f5df-47fc-a522-55e87143a70a\") " pod="kuadrant-system/authorino-5699f7bdf6-kqh58" Jul 02 20:27:47.697090 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.697068 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5699f7bdf6-kqh58" Jul 02 20:27:47.789785 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.789756 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djlfj\" (UniqueName: \"kubernetes.io/projected/03b5a35f-f5df-47fc-a522-55e87143a70a-kube-api-access-djlfj\") pod \"authorino-5699f7bdf6-kqh58\" (UID: \"03b5a35f-f5df-47fc-a522-55e87143a70a\") " pod="kuadrant-system/authorino-5699f7bdf6-kqh58" Jul 02 20:27:47.803574 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.803508 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djlfj\" (UniqueName: \"kubernetes.io/projected/03b5a35f-f5df-47fc-a522-55e87143a70a-kube-api-access-djlfj\") pod \"authorino-5699f7bdf6-kqh58\" (UID: \"03b5a35f-f5df-47fc-a522-55e87143a70a\") " pod="kuadrant-system/authorino-5699f7bdf6-kqh58" Jul 02 20:27:47.991687 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.991652 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djlfj\" (UniqueName: \"kubernetes.io/projected/03b5a35f-f5df-47fc-a522-55e87143a70a-kube-api-access-djlfj\") pod \"03b5a35f-f5df-47fc-a522-55e87143a70a\" (UID: \"03b5a35f-f5df-47fc-a522-55e87143a70a\") " Jul 02 20:27:47.993935 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:47.993911 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b5a35f-f5df-47fc-a522-55e87143a70a-kube-api-access-djlfj" (OuterVolumeSpecName: "kube-api-access-djlfj") pod "03b5a35f-f5df-47fc-a522-55e87143a70a" (UID: "03b5a35f-f5df-47fc-a522-55e87143a70a"). InnerVolumeSpecName "kube-api-access-djlfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:27:48.092330 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:48.092235 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djlfj\" (UniqueName: \"kubernetes.io/projected/03b5a35f-f5df-47fc-a522-55e87143a70a-kube-api-access-djlfj\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:27:48.694184 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:48.694149 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5699f7bdf6-kqh58" Jul 02 20:27:48.694184 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:48.694147 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-lvgfs" event={"ID":"bc3f02f3-4e88-4077-b84f-1658aab43f10","Type":"ContainerStarted","Data":"1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4"} Jul 02 20:27:48.694184 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:48.694168 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-lvgfs" podUID="bc3f02f3-4e88-4077-b84f-1658aab43f10" containerName="authorino" containerID="cri-o://1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4" gracePeriod=30 Jul 02 20:27:48.694495 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:48.694198 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-lvgfs" event={"ID":"bc3f02f3-4e88-4077-b84f-1658aab43f10","Type":"ContainerStarted","Data":"d7c283e0b00376cffa1004a5888afd15cc5fd23850d7b8a4f5f663a7612040cb"} Jul 02 20:27:48.769983 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:48.769893 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-lvgfs" podStartSLOduration=1.298472058 podStartE2EDuration="1.769876235s" totalImagesPullingTime="471.404177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:27:48.726851154 +0000 UTC m=+777.723154384" watchObservedRunningTime="2026-07-02 20:27:48.769876235 +0000 UTC m=+777.766179473" Jul 02 20:27:48.783423 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:48.783396 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5699f7bdf6-kqh58"] Jul 02 20:27:48.786801 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:48.786777 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5699f7bdf6-kqh58"] Jul 02 20:27:48.944665 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:48.944640 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lvgfs" Jul 02 20:27:49.102300 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.102195 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbplt\" (UniqueName: \"kubernetes.io/projected/bc3f02f3-4e88-4077-b84f-1658aab43f10-kube-api-access-xbplt\") pod \"bc3f02f3-4e88-4077-b84f-1658aab43f10\" (UID: \"bc3f02f3-4e88-4077-b84f-1658aab43f10\") " Jul 02 20:27:49.104325 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.104301 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3f02f3-4e88-4077-b84f-1658aab43f10-kube-api-access-xbplt" (OuterVolumeSpecName: "kube-api-access-xbplt") pod "bc3f02f3-4e88-4077-b84f-1658aab43f10" (UID: "bc3f02f3-4e88-4077-b84f-1658aab43f10"). InnerVolumeSpecName "kube-api-access-xbplt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:27:49.202778 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.202746 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbplt\" (UniqueName: \"kubernetes.io/projected/bc3f02f3-4e88-4077-b84f-1658aab43f10-kube-api-access-xbplt\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:27:49.571889 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.571857 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03b5a35f-f5df-47fc-a522-55e87143a70a" path="/var/lib/kubelet/pods/03b5a35f-f5df-47fc-a522-55e87143a70a/volumes" Jul 02 20:27:49.700630 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.700592 2565 generic.go:356] "Generic (PLEG): container finished" podID="bc3f02f3-4e88-4077-b84f-1658aab43f10" containerID="1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4" exitCode=0 Jul 02 20:27:49.700795 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.700643 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-lvgfs" Jul 02 20:27:49.700795 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.700676 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-lvgfs" event={"ID":"bc3f02f3-4e88-4077-b84f-1658aab43f10","Type":"ContainerDied","Data":"1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4"} Jul 02 20:27:49.700795 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.700723 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-lvgfs" event={"ID":"bc3f02f3-4e88-4077-b84f-1658aab43f10","Type":"ContainerDied","Data":"d7c283e0b00376cffa1004a5888afd15cc5fd23850d7b8a4f5f663a7612040cb"} Jul 02 20:27:49.700795 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.700745 2565 scope.go:117] "RemoveContainer" containerID="1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4" Jul 02 20:27:49.709105 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.709089 2565 scope.go:117] "RemoveContainer" containerID="1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4" Jul 02 20:27:49.709380 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:27:49.709362 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4\": container with ID starting with 1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4 not found: ID does not exist" containerID="1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4" Jul 02 20:27:49.709448 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.709386 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4"} err="failed to get container status \"1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4\": rpc error: code = NotFound desc = could not find container \"1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4\": container with ID starting with 1eae158c24c980ca928ef066a25ca4a1c51f89853425126140ba81315f32e6f4 not found: ID does not exist" Jul 02 20:27:49.729533 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.729509 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lvgfs"] Jul 02 20:27:49.738285 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:49.738267 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-lvgfs"] Jul 02 20:27:51.570911 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:51.570879 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc3f02f3-4e88-4077-b84f-1658aab43f10" path="/var/lib/kubelet/pods/bc3f02f3-4e88-4077-b84f-1658aab43f10/volumes" Jul 02 20:27:53.933865 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:53.933834 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:54.145660 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.145618 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:54.278575 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.278501 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7756784977-7zjlb"] Jul 02 20:27:54.278853 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.278839 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3f02f3-4e88-4077-b84f-1658aab43f10" containerName="authorino" Jul 02 20:27:54.278902 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.278856 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3f02f3-4e88-4077-b84f-1658aab43f10" containerName="authorino" Jul 02 20:27:54.278936 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.278932 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc3f02f3-4e88-4077-b84f-1658aab43f10" containerName="authorino" Jul 02 20:27:54.316137 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.316108 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:54.417354 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.417322 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7756784977-7zjlb"] Jul 02 20:27:54.417503 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.417435 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:27:54.422381 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.422356 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 02 20:27:54.422381 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.422374 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 02 20:27:54.422753 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.422598 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dv45p\"" Jul 02 20:27:54.549075 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.548992 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tzv4\" (UniqueName: \"kubernetes.io/projected/eb412d6b-a78a-4da8-a333-7923bc6afa4c-kube-api-access-2tzv4\") pod \"maas-controller-7756784977-7zjlb\" (UID: \"eb412d6b-a78a-4da8-a333-7923bc6afa4c\") " pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:27:54.549075 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.549049 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb412d6b-a78a-4da8-a333-7923bc6afa4c-webhook-cert\") pod \"maas-controller-7756784977-7zjlb\" (UID: \"eb412d6b-a78a-4da8-a333-7923bc6afa4c\") " pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:27:54.649932 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.649891 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tzv4\" (UniqueName: \"kubernetes.io/projected/eb412d6b-a78a-4da8-a333-7923bc6afa4c-kube-api-access-2tzv4\") pod \"maas-controller-7756784977-7zjlb\" (UID: \"eb412d6b-a78a-4da8-a333-7923bc6afa4c\") " pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:27:54.650113 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.649967 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb412d6b-a78a-4da8-a333-7923bc6afa4c-webhook-cert\") pod \"maas-controller-7756784977-7zjlb\" (UID: \"eb412d6b-a78a-4da8-a333-7923bc6afa4c\") " pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:27:54.652493 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.652470 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb412d6b-a78a-4da8-a333-7923bc6afa4c-webhook-cert\") pod \"maas-controller-7756784977-7zjlb\" (UID: \"eb412d6b-a78a-4da8-a333-7923bc6afa4c\") " pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:27:54.665800 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.665769 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tzv4\" (UniqueName: \"kubernetes.io/projected/eb412d6b-a78a-4da8-a333-7923bc6afa4c-kube-api-access-2tzv4\") pod \"maas-controller-7756784977-7zjlb\" (UID: \"eb412d6b-a78a-4da8-a333-7923bc6afa4c\") " pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:27:54.728086 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.728058 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:27:54.915029 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:54.915003 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7756784977-7zjlb"] Jul 02 20:27:54.917477 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:27:54.917450 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb412d6b_a78a_4da8_a333_7923bc6afa4c.slice/crio-5b36fbb59a3c9777a6c3f496ec4560376a55ccb5e94a4773e4cde137907a5d01 WatchSource:0}: Error finding container 5b36fbb59a3c9777a6c3f496ec4560376a55ccb5e94a4773e4cde137907a5d01: Status 404 returned error can't find the container with id 5b36fbb59a3c9777a6c3f496ec4560376a55ccb5e94a4773e4cde137907a5d01 Jul 02 20:27:55.732029 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:55.731982 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7756784977-7zjlb" event={"ID":"eb412d6b-a78a-4da8-a333-7923bc6afa4c","Type":"ContainerStarted","Data":"5b36fbb59a3c9777a6c3f496ec4560376a55ccb5e94a4773e4cde137907a5d01"} Jul 02 20:27:55.761512 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:55.761413 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:56.573609 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:56.573567 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:56.711306 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:56.711278 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:57.408743 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:57.408713 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:57.515931 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:57.515900 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:27:57.742054 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:57.741967 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7756784977-7zjlb" event={"ID":"eb412d6b-a78a-4da8-a333-7923bc6afa4c","Type":"ContainerStarted","Data":"d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d"} Jul 02 20:27:57.742204 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:57.742079 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:27:57.773130 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:27:57.773084 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7756784977-7zjlb" podStartSLOduration=1.9086378389999998 podStartE2EDuration="3.773070509s" totalImagesPullingTime="1.86443267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:27:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:27:57.770104787 +0000 UTC m=+786.766408022" watchObservedRunningTime="2026-07-02 20:27:57.773070509 +0000 UTC m=+786.769373743" Jul 02 20:28:08.751796 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:08.751770 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:28:31.333565 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:31.333521 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:28:31.786262 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:31.786198 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:28:31.830289 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:31.830257 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-55f547546d-456dr"] Jul 02 20:28:31.836231 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:31.836200 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:31.839516 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:31.839453 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-pz8h4\"" Jul 02 20:28:31.840541 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:31.840518 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 02 20:28:31.851850 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:31.851826 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-55f547546d-456dr"] Jul 02 20:28:31.881561 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:31.881535 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:28:31.981110 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:31.981074 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7c2b51-b00f-41e8-92d3-d97e2e13d53e-openshift-service-ca\") pod \"maas-api-55f547546d-456dr\" (UID: \"5a7c2b51-b00f-41e8-92d3-d97e2e13d53e\") " pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:31.981307 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:31.981139 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7c2b51-b00f-41e8-92d3-d97e2e13d53e-maas-api-tls\") pod \"maas-api-55f547546d-456dr\" (UID: \"5a7c2b51-b00f-41e8-92d3-d97e2e13d53e\") " pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:31.981307 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:31.981176 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6nlq\" (UniqueName: \"kubernetes.io/projected/5a7c2b51-b00f-41e8-92d3-d97e2e13d53e-kube-api-access-z6nlq\") pod \"maas-api-55f547546d-456dr\" (UID: \"5a7c2b51-b00f-41e8-92d3-d97e2e13d53e\") " pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:32.082644 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.082561 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7c2b51-b00f-41e8-92d3-d97e2e13d53e-maas-api-tls\") pod \"maas-api-55f547546d-456dr\" (UID: \"5a7c2b51-b00f-41e8-92d3-d97e2e13d53e\") " pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:32.082918 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.082890 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6nlq\" (UniqueName: \"kubernetes.io/projected/5a7c2b51-b00f-41e8-92d3-d97e2e13d53e-kube-api-access-z6nlq\") pod \"maas-api-55f547546d-456dr\" (UID: \"5a7c2b51-b00f-41e8-92d3-d97e2e13d53e\") " pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:32.083031 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.082989 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7c2b51-b00f-41e8-92d3-d97e2e13d53e-openshift-service-ca\") pod \"maas-api-55f547546d-456dr\" (UID: \"5a7c2b51-b00f-41e8-92d3-d97e2e13d53e\") " pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:32.083673 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.083650 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a7c2b51-b00f-41e8-92d3-d97e2e13d53e-openshift-service-ca\") pod \"maas-api-55f547546d-456dr\" (UID: \"5a7c2b51-b00f-41e8-92d3-d97e2e13d53e\") " pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:32.085465 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.085443 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a7c2b51-b00f-41e8-92d3-d97e2e13d53e-maas-api-tls\") pod \"maas-api-55f547546d-456dr\" (UID: \"5a7c2b51-b00f-41e8-92d3-d97e2e13d53e\") " pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:32.095511 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.095486 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6nlq\" (UniqueName: \"kubernetes.io/projected/5a7c2b51-b00f-41e8-92d3-d97e2e13d53e-kube-api-access-z6nlq\") pod \"maas-api-55f547546d-456dr\" (UID: \"5a7c2b51-b00f-41e8-92d3-d97e2e13d53e\") " pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:32.147002 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.146968 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:32.247469 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.247442 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:28:32.287791 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.287765 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-55f547546d-456dr"] Jul 02 20:28:32.290601 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:28:32.290564 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a7c2b51_b00f_41e8_92d3_d97e2e13d53e.slice/crio-ac4adc84a5e3244c0472ffa5e8d1fc148023dba3653738aabe2a0e54f01744f2 WatchSource:0}: Error finding container ac4adc84a5e3244c0472ffa5e8d1fc148023dba3653738aabe2a0e54f01744f2: Status 404 returned error can't find the container with id ac4adc84a5e3244c0472ffa5e8d1fc148023dba3653738aabe2a0e54f01744f2 Jul 02 20:28:32.356982 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.356901 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:28:32.379409 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.379379 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-l6qqw"] Jul 02 20:28:32.384868 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.384843 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" Jul 02 20:28:32.388976 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.388951 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 02 20:28:32.389117 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.389098 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rpsch\"" Jul 02 20:28:32.395405 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.394918 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-l6qqw"] Jul 02 20:28:32.486148 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.486117 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77a61db-6c15-41dd-8d9b-a1e8aac0d648-plugins-config-volume\") pod \"payload-processing-58f86c7798-l6qqw\" (UID: \"e77a61db-6c15-41dd-8d9b-a1e8aac0d648\") " pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" Jul 02 20:28:32.486148 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.486153 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4td2m\" (UniqueName: \"kubernetes.io/projected/e77a61db-6c15-41dd-8d9b-a1e8aac0d648-kube-api-access-4td2m\") pod \"payload-processing-58f86c7798-l6qqw\" (UID: \"e77a61db-6c15-41dd-8d9b-a1e8aac0d648\") " pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" Jul 02 20:28:32.587249 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.587189 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77a61db-6c15-41dd-8d9b-a1e8aac0d648-plugins-config-volume\") pod \"payload-processing-58f86c7798-l6qqw\" (UID: \"e77a61db-6c15-41dd-8d9b-a1e8aac0d648\") " pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" Jul 02 20:28:32.587417 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.587259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4td2m\" (UniqueName: \"kubernetes.io/projected/e77a61db-6c15-41dd-8d9b-a1e8aac0d648-kube-api-access-4td2m\") pod \"payload-processing-58f86c7798-l6qqw\" (UID: \"e77a61db-6c15-41dd-8d9b-a1e8aac0d648\") " pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" Jul 02 20:28:32.587902 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.587882 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e77a61db-6c15-41dd-8d9b-a1e8aac0d648-plugins-config-volume\") pod \"payload-processing-58f86c7798-l6qqw\" (UID: \"e77a61db-6c15-41dd-8d9b-a1e8aac0d648\") " pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" Jul 02 20:28:32.599400 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.599372 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4td2m\" (UniqueName: \"kubernetes.io/projected/e77a61db-6c15-41dd-8d9b-a1e8aac0d648-kube-api-access-4td2m\") pod \"payload-processing-58f86c7798-l6qqw\" (UID: \"e77a61db-6c15-41dd-8d9b-a1e8aac0d648\") " pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" Jul 02 20:28:32.699398 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.699365 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" Jul 02 20:28:32.840994 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.840943 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-l6qqw"] Jul 02 20:28:32.844533 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:28:32.844491 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77a61db_6c15_41dd_8d9b_a1e8aac0d648.slice/crio-f915907aed0d520497cf8e13218136810f0d4b392dc374c2ca6f00ac703f0d9d WatchSource:0}: Error finding container f915907aed0d520497cf8e13218136810f0d4b392dc374c2ca6f00ac703f0d9d: Status 404 returned error can't find the container with id f915907aed0d520497cf8e13218136810f0d4b392dc374c2ca6f00ac703f0d9d Jul 02 20:28:32.907914 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.907871 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" event={"ID":"e77a61db-6c15-41dd-8d9b-a1e8aac0d648","Type":"ContainerStarted","Data":"f915907aed0d520497cf8e13218136810f0d4b392dc374c2ca6f00ac703f0d9d"} Jul 02 20:28:32.909098 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:32.909068 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-55f547546d-456dr" event={"ID":"5a7c2b51-b00f-41e8-92d3-d97e2e13d53e","Type":"ContainerStarted","Data":"ac4adc84a5e3244c0472ffa5e8d1fc148023dba3653738aabe2a0e54f01744f2"} Jul 02 20:28:33.916348 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:33.916312 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-55f547546d-456dr" event={"ID":"5a7c2b51-b00f-41e8-92d3-d97e2e13d53e","Type":"ContainerStarted","Data":"82cc6e11bd0d091b3d30d55f1f485d22c577569dfa64ad6e8ce203a5a9bc604c"} Jul 02 20:28:33.916883 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:33.916567 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:33.937097 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:33.937044 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-55f547546d-456dr" podStartSLOduration=1.665192132 podStartE2EDuration="2.937029542s" totalImagesPullingTime="1.27183741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:28:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:28:33.936139175 +0000 UTC m=+822.932442408" watchObservedRunningTime="2026-07-02 20:28:33.937029542 +0000 UTC m=+822.933332775" Jul 02 20:28:34.921648 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:34.921616 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" event={"ID":"e77a61db-6c15-41dd-8d9b-a1e8aac0d648","Type":"ContainerStarted","Data":"a8bb20add64aaf14dc479e07da5ec75976c6d52e4b7242e6a08de18ebf78e4d9"} Jul 02 20:28:34.922032 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:34.921928 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" Jul 02 20:28:34.940060 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:34.940013 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" podStartSLOduration=1.488660157 podStartE2EDuration="2.939998489s" totalImagesPullingTime="1.451338332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:28:34.938261558 +0000 UTC m=+823.934564790" watchObservedRunningTime="2026-07-02 20:28:34.939998489 +0000 UTC m=+823.936301721" Jul 02 20:28:39.926842 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:39.926814 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-55f547546d-456dr" Jul 02 20:28:40.935339 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:40.935300 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:28:41.025745 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:41.025713 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:28:45.929304 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:45.929268 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-l6qqw" Jul 02 20:28:47.271991 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:47.271956 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:28:57.559087 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:28:57.559052 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:29:09.399604 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.399571 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2"] Jul 02 20:29:09.406579 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.406557 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.410252 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.410195 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 02 20:29:09.410398 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.410272 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 02 20:29:09.410398 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.410229 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 02 20:29:09.410615 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.410581 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xgxfv\"" Jul 02 20:29:09.411156 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.411135 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2"] Jul 02 20:29:09.435013 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.434984 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:29:09.516508 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.516450 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.516672 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.516516 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.516672 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.516625 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjpmb\" (UniqueName: \"kubernetes.io/projected/43463c81-11aa-4a42-bfb0-b55064428a4c-kube-api-access-gjpmb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.516672 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.516669 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.516782 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.516766 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43463c81-11aa-4a42-bfb0-b55064428a4c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.516817 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.516793 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.617911 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.617875 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.618107 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.617930 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjpmb\" (UniqueName: \"kubernetes.io/projected/43463c81-11aa-4a42-bfb0-b55064428a4c-kube-api-access-gjpmb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.618107 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.617955 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.618107 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.617995 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43463c81-11aa-4a42-bfb0-b55064428a4c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.618107 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.618014 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.618107 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.618030 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.618448 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.618379 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.618507 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.618445 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.618507 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.618493 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.620516 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.620489 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43463c81-11aa-4a42-bfb0-b55064428a4c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.620683 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.620668 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43463c81-11aa-4a42-bfb0-b55064428a4c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.628929 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.628905 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjpmb\" (UniqueName: \"kubernetes.io/projected/43463c81-11aa-4a42-bfb0-b55064428a4c-kube-api-access-gjpmb\") pod \"e2e-distinct-simulated-kserve-59975d69f9-krgm2\" (UID: \"43463c81-11aa-4a42-bfb0-b55064428a4c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.719344 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.719252 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:09.855992 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:09.855969 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2"] Jul 02 20:29:09.858055 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:29:09.858027 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43463c81_11aa_4a42_bfb0_b55064428a4c.slice/crio-ecd9e2c66f59a50908da477b32c3b587a751015e44aebab47e2b76b5183bf617 WatchSource:0}: Error finding container ecd9e2c66f59a50908da477b32c3b587a751015e44aebab47e2b76b5183bf617: Status 404 returned error can't find the container with id ecd9e2c66f59a50908da477b32c3b587a751015e44aebab47e2b76b5183bf617 Jul 02 20:29:10.087050 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:10.086961 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" event={"ID":"43463c81-11aa-4a42-bfb0-b55064428a4c","Type":"ContainerStarted","Data":"ecd9e2c66f59a50908da477b32c3b587a751015e44aebab47e2b76b5183bf617"} Jul 02 20:29:11.093167 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:11.093134 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" event={"ID":"43463c81-11aa-4a42-bfb0-b55064428a4c","Type":"ContainerStarted","Data":"9739bc408799cfc6b5ded8891d54c7762965835574eb0893e71681e93a52cae3"} Jul 02 20:29:11.131981 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:11.131929 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" podStartSLOduration=1.093268804 podStartE2EDuration="2.131913549s" totalImagesPullingTime="1.038644745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:29:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:29:11.12974269 +0000 UTC m=+860.126045923" watchObservedRunningTime="2026-07-02 20:29:11.131913549 +0000 UTC m=+860.128216782" Jul 02 20:29:19.251616 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:19.251566 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:29:19.719558 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:19.719524 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:19.719732 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:19.719610 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:19.731786 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:19.731759 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:20.147979 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:20.147955 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-krgm2" Jul 02 20:29:22.329130 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:22.329095 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:29:31.009096 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.009060 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q"] Jul 02 20:29:31.012916 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.012893 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.015742 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.015718 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 02 20:29:31.027043 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.027018 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q"] Jul 02 20:29:31.048388 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.048358 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:29:31.113696 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.113657 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dzwb\" (UniqueName: \"kubernetes.io/projected/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-kube-api-access-4dzwb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.113696 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.113698 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.113914 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.113718 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.113914 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.113789 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.113914 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.113826 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.113914 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.113871 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.215286 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.215245 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dzwb\" (UniqueName: \"kubernetes.io/projected/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-kube-api-access-4dzwb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.215286 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.215288 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.215540 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.215306 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.215540 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.215324 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.215540 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.215429 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.215540 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.215476 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.215725 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.215706 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.215820 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.215797 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.215868 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.215836 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.217683 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.217656 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.217972 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.217956 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.229018 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.228994 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dzwb\" (UniqueName: \"kubernetes.io/projected/d8fce5e7-9a3e-4a14-a868-ffedc13b976e-kube-api-access-4dzwb\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-t6h5q\" (UID: \"d8fce5e7-9a3e-4a14-a868-ffedc13b976e\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.323477 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.323385 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:31.465417 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:31.465390 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q"] Jul 02 20:29:31.467855 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:29:31.467826 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fce5e7_9a3e_4a14_a868_ffedc13b976e.slice/crio-eb53a4475b085aaca802ee070766c190246517b2110f8f5ac9250d135942fe67 WatchSource:0}: Error finding container eb53a4475b085aaca802ee070766c190246517b2110f8f5ac9250d135942fe67: Status 404 returned error can't find the container with id eb53a4475b085aaca802ee070766c190246517b2110f8f5ac9250d135942fe67 Jul 02 20:29:32.195013 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:32.194978 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" event={"ID":"d8fce5e7-9a3e-4a14-a868-ffedc13b976e","Type":"ContainerStarted","Data":"341c3ee35cde70dc0038b72ea368084fbfe7bd0630523c87c934f990f8945650"} Jul 02 20:29:32.195424 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:32.195042 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" event={"ID":"d8fce5e7-9a3e-4a14-a868-ffedc13b976e","Type":"ContainerStarted","Data":"eb53a4475b085aaca802ee070766c190246517b2110f8f5ac9250d135942fe67"} Jul 02 20:29:32.216533 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:32.216479 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" podStartSLOduration=2.040962038 podStartE2EDuration="2.216465412s" totalImagesPullingTime="175.503374ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:29:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:29:32.215384071 +0000 UTC m=+881.211687306" watchObservedRunningTime="2026-07-02 20:29:32.216465412 +0000 UTC m=+881.212768650" Jul 02 20:29:38.595145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.595106 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5"] Jul 02 20:29:38.599241 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.599200 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.601739 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.601705 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 02 20:29:38.608163 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.608136 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5"] Jul 02 20:29:38.634494 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.634463 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:29:38.685288 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.685259 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.685406 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.685295 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6cd414b-430d-471f-9a9e-57f98988d182-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.685406 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.685366 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h8qb\" (UniqueName: \"kubernetes.io/projected/a6cd414b-430d-471f-9a9e-57f98988d182-kube-api-access-6h8qb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.685480 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.685401 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.685630 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.685611 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.685724 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.685706 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.786415 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.786370 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.786415 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.786415 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6cd414b-430d-471f-9a9e-57f98988d182-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.786680 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.786451 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h8qb\" (UniqueName: \"kubernetes.io/projected/a6cd414b-430d-471f-9a9e-57f98988d182-kube-api-access-6h8qb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.786680 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.786478 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.786680 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.786558 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.786680 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.786611 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.786867 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.786824 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.786986 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.786967 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.787041 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.786983 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.788938 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.788914 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6cd414b-430d-471f-9a9e-57f98988d182-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.789167 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.789149 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6cd414b-430d-471f-9a9e-57f98988d182-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.795056 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.795034 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h8qb\" (UniqueName: \"kubernetes.io/projected/a6cd414b-430d-471f-9a9e-57f98988d182-kube-api-access-6h8qb\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5\" (UID: \"a6cd414b-430d-471f-9a9e-57f98988d182\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:38.931079 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:38.931046 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:39.070538 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:39.070512 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5"] Jul 02 20:29:39.072292 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:29:39.072263 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6cd414b_430d_471f_9a9e_57f98988d182.slice/crio-4ca323f771deea634ed8d24df358f1a2283aecd1cc7932ec0bc8c9311d54da54 WatchSource:0}: Error finding container 4ca323f771deea634ed8d24df358f1a2283aecd1cc7932ec0bc8c9311d54da54: Status 404 returned error can't find the container with id 4ca323f771deea634ed8d24df358f1a2283aecd1cc7932ec0bc8c9311d54da54 Jul 02 20:29:39.229554 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:39.229476 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" event={"ID":"a6cd414b-430d-471f-9a9e-57f98988d182","Type":"ContainerStarted","Data":"4ca323f771deea634ed8d24df358f1a2283aecd1cc7932ec0bc8c9311d54da54"} Jul 02 20:29:40.236000 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:40.235966 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" event={"ID":"a6cd414b-430d-471f-9a9e-57f98988d182","Type":"ContainerStarted","Data":"4d175dcf4cc7c5dc2d32098346fcb2f7de4e372638f9acce52fde1e0f588fe44"} Jul 02 20:29:40.259404 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:40.259343 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" podStartSLOduration=2.076380843 podStartE2EDuration="2.259321319s" totalImagesPullingTime="182.940476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:29:40.257812413 +0000 UTC m=+889.254115648" watchObservedRunningTime="2026-07-02 20:29:40.259321319 +0000 UTC m=+889.255624556" Jul 02 20:29:41.323742 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:41.323702 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:41.324270 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:41.323776 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:41.336418 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:41.336392 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:42.257274 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:42.257243 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-t6h5q" Jul 02 20:29:48.931649 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:48.931606 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:48.932048 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:48.931853 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:48.944170 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:48.944144 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:49.295988 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:49.295895 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5" Jul 02 20:29:51.474457 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:51.474433 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:29:51.475355 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:29:51.475336 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:30:29.765989 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:30:29.765955 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:30:29.865458 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:30:29.865429 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:30:29.966447 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:30:29.966417 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:30:31.071905 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:30:31.071871 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:30:31.154526 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:30:31.154471 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:31:13.556646 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:13.556568 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:31:34.332817 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:34.332787 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:31:34.393347 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:34.393318 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:31:49.074438 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.074396 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:31:49.203980 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.203946 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7756784977-7zjlb"] Jul 02 20:31:49.204286 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.204254 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7756784977-7zjlb" podUID="eb412d6b-a78a-4da8-a333-7923bc6afa4c" containerName="manager" containerID="cri-o://d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d" gracePeriod=30 Jul 02 20:31:49.455394 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.455366 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:31:49.553326 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.553295 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tzv4\" (UniqueName: \"kubernetes.io/projected/eb412d6b-a78a-4da8-a333-7923bc6afa4c-kube-api-access-2tzv4\") pod \"eb412d6b-a78a-4da8-a333-7923bc6afa4c\" (UID: \"eb412d6b-a78a-4da8-a333-7923bc6afa4c\") " Jul 02 20:31:49.553326 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.553328 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb412d6b-a78a-4da8-a333-7923bc6afa4c-webhook-cert\") pod \"eb412d6b-a78a-4da8-a333-7923bc6afa4c\" (UID: \"eb412d6b-a78a-4da8-a333-7923bc6afa4c\") " Jul 02 20:31:49.555675 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.555648 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb412d6b-a78a-4da8-a333-7923bc6afa4c-kube-api-access-2tzv4" (OuterVolumeSpecName: "kube-api-access-2tzv4") pod "eb412d6b-a78a-4da8-a333-7923bc6afa4c" (UID: "eb412d6b-a78a-4da8-a333-7923bc6afa4c"). InnerVolumeSpecName "kube-api-access-2tzv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:31:49.555809 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.555790 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb412d6b-a78a-4da8-a333-7923bc6afa4c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "eb412d6b-a78a-4da8-a333-7923bc6afa4c" (UID: "eb412d6b-a78a-4da8-a333-7923bc6afa4c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:31:49.654146 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.654113 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tzv4\" (UniqueName: \"kubernetes.io/projected/eb412d6b-a78a-4da8-a333-7923bc6afa4c-kube-api-access-2tzv4\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:31:49.654146 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.654144 2565 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb412d6b-a78a-4da8-a333-7923bc6afa4c-webhook-cert\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:31:49.867594 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.867560 2565 generic.go:356] "Generic (PLEG): container finished" podID="eb412d6b-a78a-4da8-a333-7923bc6afa4c" containerID="d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d" exitCode=0 Jul 02 20:31:49.867594 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.867598 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7756784977-7zjlb" Jul 02 20:31:49.867832 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.867656 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7756784977-7zjlb" event={"ID":"eb412d6b-a78a-4da8-a333-7923bc6afa4c","Type":"ContainerDied","Data":"d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d"} Jul 02 20:31:49.867832 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.867707 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7756784977-7zjlb" event={"ID":"eb412d6b-a78a-4da8-a333-7923bc6afa4c","Type":"ContainerDied","Data":"5b36fbb59a3c9777a6c3f496ec4560376a55ccb5e94a4773e4cde137907a5d01"} Jul 02 20:31:49.867832 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.867737 2565 scope.go:117] "RemoveContainer" containerID="d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d" Jul 02 20:31:49.876382 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.876364 2565 scope.go:117] "RemoveContainer" containerID="d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d" Jul 02 20:31:49.876657 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:31:49.876640 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d\": container with ID starting with d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d not found: ID does not exist" containerID="d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d" Jul 02 20:31:49.876711 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.876664 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d"} err="failed to get container status \"d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d\": rpc error: code = NotFound desc = could not find container \"d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d\": container with ID starting with d376b9352ab7c0bfa5cf9e7dcf0c6ac727793f6181ad8b03151d46144784961d not found: ID does not exist" Jul 02 20:31:49.884629 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.884604 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7756784977-7zjlb"] Jul 02 20:31:49.890273 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:49.890252 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7756784977-7zjlb"] Jul 02 20:31:50.575965 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:50.575936 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:31:51.571410 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:31:51.571379 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb412d6b-a78a-4da8-a333-7923bc6afa4c" path="/var/lib/kubelet/pods/eb412d6b-a78a-4da8-a333-7923bc6afa4c/volumes" Jul 02 20:32:03.546063 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:32:03.546011 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:32:03.613119 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:32:03.613085 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:32:25.880764 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:32:25.880731 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:32:25.981195 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:32:25.981149 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:32:26.176444 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:32:26.176417 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:32:32.099756 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:32:32.099710 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:32:32.201858 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:32:32.201815 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:32:32.300596 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:32:32.300563 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:32:32.618276 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:32:32.618239 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:32:38.635274 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:32:38.635240 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:32:38.736881 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:32:38.736843 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:33:13.748918 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:33:13.748865 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:33:21.226357 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:33:21.226315 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:34:03.388408 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:34:03.388359 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:34:51.501050 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:34:51.501020 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:34:51.503470 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:34:51.503453 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:39:19.664466 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:39:19.664425 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:39:20.945124 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:39:20.945081 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:39:42.865660 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:39:42.865626 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:39:51.530116 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:39:51.530089 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:39:51.532840 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:39:51.532800 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:40:49.475374 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:40:49.475332 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:43:36.201585 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:43:36.201547 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:43:43.670132 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:43:43.670086 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:43:51.941322 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:43:51.941292 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:43:57.040971 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:43:57.040936 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:43:59.614272 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:43:59.614236 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:43:59.866491 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:43:59.866405 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:43:59.963633 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:43:59.963605 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:00.338935 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:00.338903 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:06.046717 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:06.046681 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:06.150536 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:06.150481 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:06.256694 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:06.256668 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:06.321621 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:06.321542 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:07.575667 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:07.575643 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:07.833353 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:07.833277 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:08.065193 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:08.065164 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:08.162487 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:08.162459 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:08.462991 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:08.462909 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:08.581393 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:08.581365 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:10.034923 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:10.034892 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:12.570987 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:12.570952 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:12.758305 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:12.758275 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:12.861649 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:12.861546 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:12.963502 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:12.963475 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:14.619842 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:14.619817 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:18.752023 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:18.751995 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:18.848956 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:18.848928 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:19.211752 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:19.211721 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:19.277307 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:19.277282 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:20.266765 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.266721 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:20.388275 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.388242 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6"] Jul 02 20:44:20.388677 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.388661 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb412d6b-a78a-4da8-a333-7923bc6afa4c" containerName="manager" Jul 02 20:44:20.388769 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.388680 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb412d6b-a78a-4da8-a333-7923bc6afa4c" containerName="manager" Jul 02 20:44:20.388822 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.388794 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb412d6b-a78a-4da8-a333-7923bc6afa4c" containerName="manager" Jul 02 20:44:20.395619 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.395593 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:20.398405 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.398385 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-6eb4b1ad-serving-cert\"" Jul 02 20:44:20.406580 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.406559 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6"] Jul 02 20:44:20.409096 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.409071 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:20.468701 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.468665 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k78bv\" (UniqueName: \"kubernetes.io/projected/35f68e66-bb02-4dfb-a008-7ac8f4864107-kube-api-access-k78bv\") pod \"maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:20.468701 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.468711 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35f68e66-bb02-4dfb-a008-7ac8f4864107-maas-api-tls\") pod \"maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:20.468914 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.468801 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f68e66-bb02-4dfb-a008-7ac8f4864107-openshift-service-ca\") pod \"maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:20.470156 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.470132 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:20.570168 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.570080 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f68e66-bb02-4dfb-a008-7ac8f4864107-openshift-service-ca\") pod \"maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:20.570168 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.570136 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k78bv\" (UniqueName: \"kubernetes.io/projected/35f68e66-bb02-4dfb-a008-7ac8f4864107-kube-api-access-k78bv\") pod \"maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:20.570438 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.570255 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35f68e66-bb02-4dfb-a008-7ac8f4864107-maas-api-tls\") pod \"maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:20.570438 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:44:20.570420 2565 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-ait-6eb4b1ad-serving-cert: secret "maas-api-e2e-ait-6eb4b1ad-serving-cert" not found Jul 02 20:44:20.570550 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:44:20.570510 2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35f68e66-bb02-4dfb-a008-7ac8f4864107-maas-api-tls podName:35f68e66-bb02-4dfb-a008-7ac8f4864107 nodeName:}" failed. No retries permitted until 2026-07-02 20:44:21.070486063 +0000 UTC m=+1770.066789290 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/35f68e66-bb02-4dfb-a008-7ac8f4864107-maas-api-tls") pod "maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" (UID: "35f68e66-bb02-4dfb-a008-7ac8f4864107") : secret "maas-api-e2e-ait-6eb4b1ad-serving-cert" not found Jul 02 20:44:20.570841 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.570819 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f68e66-bb02-4dfb-a008-7ac8f4864107-openshift-service-ca\") pod \"maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:20.580618 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.580594 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k78bv\" (UniqueName: \"kubernetes.io/projected/35f68e66-bb02-4dfb-a008-7ac8f4864107-kube-api-access-k78bv\") pod \"maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:20.737168 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.737138 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:20.896701 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:20.896617 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:21.075398 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:21.075367 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35f68e66-bb02-4dfb-a008-7ac8f4864107-maas-api-tls\") pod \"maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:21.077881 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:21.077852 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35f68e66-bb02-4dfb-a008-7ac8f4864107-maas-api-tls\") pod \"maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:21.310044 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:21.310002 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:21.446258 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:21.446200 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6"] Jul 02 20:44:21.448161 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:44:21.448137 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f68e66_bb02_4dfb_a008_7ac8f4864107.slice/crio-f9d2fcccd80fab08aa82c3a2443e8b7df1ed8aa03a14261790bc9f4c22d75c3c WatchSource:0}: Error finding container f9d2fcccd80fab08aa82c3a2443e8b7df1ed8aa03a14261790bc9f4c22d75c3c: Status 404 returned error can't find the container with id f9d2fcccd80fab08aa82c3a2443e8b7df1ed8aa03a14261790bc9f4c22d75c3c Jul 02 20:44:21.449854 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:21.449838 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:44:22.405987 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:22.405954 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" event={"ID":"35f68e66-bb02-4dfb-a008-7ac8f4864107","Type":"ContainerStarted","Data":"c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39"} Jul 02 20:44:22.406384 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:22.406001 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" event={"ID":"35f68e66-bb02-4dfb-a008-7ac8f4864107","Type":"ContainerStarted","Data":"f9d2fcccd80fab08aa82c3a2443e8b7df1ed8aa03a14261790bc9f4c22d75c3c"} Jul 02 20:44:22.406384 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:22.406084 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:22.424005 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:22.423956 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" podStartSLOduration=2.07415722 podStartE2EDuration="2.423942541s" totalImagesPullingTime="349.785321ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:44:22.422503382 +0000 UTC m=+1771.418806618" watchObservedRunningTime="2026-07-02 20:44:22.423942541 +0000 UTC m=+1771.420245772" Jul 02 20:44:23.492247 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:23.492195 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:25.404083 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:25.404051 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:25.789931 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:25.789852 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6"] Jul 02 20:44:25.790082 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:25.790019 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" podUID="35f68e66-bb02-4dfb-a008-7ac8f4864107" containerName="maas-api" containerID="cri-o://c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39" gracePeriod=30 Jul 02 20:44:25.819025 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:25.819000 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:26.035796 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.035762 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:26.122531 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.122460 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k78bv\" (UniqueName: \"kubernetes.io/projected/35f68e66-bb02-4dfb-a008-7ac8f4864107-kube-api-access-k78bv\") pod \"35f68e66-bb02-4dfb-a008-7ac8f4864107\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " Jul 02 20:44:26.122531 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.122499 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35f68e66-bb02-4dfb-a008-7ac8f4864107-maas-api-tls\") pod \"35f68e66-bb02-4dfb-a008-7ac8f4864107\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " Jul 02 20:44:26.122531 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.122522 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f68e66-bb02-4dfb-a008-7ac8f4864107-openshift-service-ca\") pod \"35f68e66-bb02-4dfb-a008-7ac8f4864107\" (UID: \"35f68e66-bb02-4dfb-a008-7ac8f4864107\") " Jul 02 20:44:26.122912 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.122891 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35f68e66-bb02-4dfb-a008-7ac8f4864107-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "35f68e66-bb02-4dfb-a008-7ac8f4864107" (UID: "35f68e66-bb02-4dfb-a008-7ac8f4864107"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 20:44:26.124530 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.124510 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35f68e66-bb02-4dfb-a008-7ac8f4864107-kube-api-access-k78bv" (OuterVolumeSpecName: "kube-api-access-k78bv") pod "35f68e66-bb02-4dfb-a008-7ac8f4864107" (UID: "35f68e66-bb02-4dfb-a008-7ac8f4864107"). InnerVolumeSpecName "kube-api-access-k78bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:44:26.124694 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.124679 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35f68e66-bb02-4dfb-a008-7ac8f4864107-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "35f68e66-bb02-4dfb-a008-7ac8f4864107" (UID: "35f68e66-bb02-4dfb-a008-7ac8f4864107"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:44:26.223684 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.223657 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k78bv\" (UniqueName: \"kubernetes.io/projected/35f68e66-bb02-4dfb-a008-7ac8f4864107-kube-api-access-k78bv\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:44:26.223684 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.223682 2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35f68e66-bb02-4dfb-a008-7ac8f4864107-maas-api-tls\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:44:26.223684 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.223691 2565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35f68e66-bb02-4dfb-a008-7ac8f4864107-openshift-service-ca\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:44:26.425035 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.425001 2565 generic.go:356] "Generic (PLEG): container finished" podID="35f68e66-bb02-4dfb-a008-7ac8f4864107" containerID="c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39" exitCode=0 Jul 02 20:44:26.425507 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.425046 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" Jul 02 20:44:26.425507 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.425089 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" event={"ID":"35f68e66-bb02-4dfb-a008-7ac8f4864107","Type":"ContainerDied","Data":"c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39"} Jul 02 20:44:26.425507 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.425178 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6" event={"ID":"35f68e66-bb02-4dfb-a008-7ac8f4864107","Type":"ContainerDied","Data":"f9d2fcccd80fab08aa82c3a2443e8b7df1ed8aa03a14261790bc9f4c22d75c3c"} Jul 02 20:44:26.425507 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.425224 2565 scope.go:117] "RemoveContainer" containerID="c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39" Jul 02 20:44:26.434701 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.434684 2565 scope.go:117] "RemoveContainer" containerID="c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39" Jul 02 20:44:26.434969 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:44:26.434948 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39\": container with ID starting with c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39 not found: ID does not exist" containerID="c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39" Jul 02 20:44:26.435035 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.434981 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39"} err="failed to get container status \"c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39\": rpc error: code = NotFound desc = could not find container \"c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39\": container with ID starting with c6491c42377ac8cef8dfe3ce9358e4f4e9d29708512768282172b606e8c61f39 not found: ID does not exist" Jul 02 20:44:26.447265 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.447236 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6"] Jul 02 20:44:26.453230 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:26.453194 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-6eb4b1ad-68dd4545f6-btkt6"] Jul 02 20:44:27.571874 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:27.571843 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35f68e66-bb02-4dfb-a008-7ac8f4864107" path="/var/lib/kubelet/pods/35f68e66-bb02-4dfb-a008-7ac8f4864107/volumes" Jul 02 20:44:31.725894 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:31.725859 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:31.826196 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:31.826165 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:33.927692 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:33.927663 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:37.682455 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:37.682425 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:37.748488 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:37.748462 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:37.903175 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:37.903148 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:38.003376 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:38.003297 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:39.083611 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:39.083562 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:39.183754 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:39.183727 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:39.317013 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:39.316980 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:39.499229 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:39.499178 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:39.778572 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:39.778493 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:39.888388 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:39.888357 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:41.226574 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:41.226539 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:43.529845 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:43.529763 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:43.713280 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:43.713240 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:43.821464 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:43.821384 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:43.882255 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:43.882226 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:45.311450 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:45.311399 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:49.485858 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:49.485811 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:49.545186 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:49.545156 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:49.985087 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:49.985056 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:50.082672 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:50.082629 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:50.181773 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:50.181728 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:51.533318 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:51.533284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:51.563133 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:51.563109 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:44:51.569870 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:51.569834 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:44:57.626302 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:57.626263 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:57.728283 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:57.728254 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:57.826901 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:57.826872 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:57.936491 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:57.936460 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:44:58.356975 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:44:58.356896 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:45:24.339524 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:45:24.339467 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:45:30.069811 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:45:30.069778 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:45:30.177384 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:45:30.177354 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:45:30.270118 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:45:30.270087 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:45:46.228195 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:45:46.228164 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:45:52.153523 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:45:52.153488 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:45:52.272188 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:45:52.272157 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:45:52.380241 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:45:52.380193 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:02.918038 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:02.918004 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:03.700271 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:03.700243 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:09.480627 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:09.480594 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:09.586345 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:09.586313 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:09.681111 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:09.681082 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:09.781362 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:09.781280 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:10.576500 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:10.576472 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:10.677110 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:10.677075 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:10.814697 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:10.814668 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:12.491320 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:12.491238 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:18.334583 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:18.334533 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:18.408088 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:18.408054 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:19.085050 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:19.085009 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:25.972677 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:25.972636 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:26.055758 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:26.055726 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:26.220530 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:26.220499 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:26.545706 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:26.545664 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:27.160068 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:27.160031 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:29.500444 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:29.500414 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:29.598721 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:29.598690 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:29.732363 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:29.732336 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:30.089620 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:30.089586 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:30.252220 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:30.252189 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:30.354880 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:30.354802 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:34.003996 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:34.003969 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:34.666280 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:34.666251 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:34.767554 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:34.767525 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:35.121230 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:35.121128 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:38.880621 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:38.880588 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:38.981699 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:38.981670 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:40.946341 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:40.946308 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:41.016271 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:41.016203 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:41.357723 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:41.357634 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:41.678322 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:41.678283 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:41.778453 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:41.778422 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:47.468922 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:47.468889 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:47.533913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:47.533883 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:47.737871 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:47.737780 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:47.974687 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:47.974659 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:48.098332 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:48.098245 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:48.191037 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:48.191002 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:48.310036 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:48.310005 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:53.930999 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:53.930966 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:54.041108 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:54.041073 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:54.131528 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:54.131498 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:54.287790 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:54.287694 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:59.861416 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:59.861373 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:46:59.961615 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:46:59.961579 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:06.291558 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:06.291505 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:06.607390 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:06.607299 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:06.885796 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:06.885694 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:09.689657 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:09.689626 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:09.771053 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:09.771027 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:09.832353 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:09.832324 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:09.932873 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:09.932825 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:10.244537 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:10.244470 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:10.355408 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:10.355374 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:11.904776 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:11.904744 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:12.004462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:12.004427 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:14.182601 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:14.182570 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:26.911604 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:26.911563 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v"] Jul 02 20:47:26.911980 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:26.911952 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35f68e66-bb02-4dfb-a008-7ac8f4864107" containerName="maas-api" Jul 02 20:47:26.911980 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:26.911964 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="35f68e66-bb02-4dfb-a008-7ac8f4864107" containerName="maas-api" Jul 02 20:47:26.912060 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:26.912043 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="35f68e66-bb02-4dfb-a008-7ac8f4864107" containerName="maas-api" Jul 02 20:47:26.915041 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:26.915023 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:26.919293 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:26.919268 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-13937e3a\"/\"default-dockercfg-hzk8v\"" Jul 02 20:47:26.919423 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:26.919294 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-13937e3a\"/\"openshift-service-ca.crt\"" Jul 02 20:47:26.919423 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:26.919327 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-13937e3a\"/\"kube-root-ca.crt\"" Jul 02 20:47:26.919423 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:26.919340 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-13937e3a\"/\"e2e-lifecycle-model-13937e3a-kserve-self-signed-certs\"" Jul 02 20:47:26.924823 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:26.924804 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v"] Jul 02 20:47:26.941684 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:26.941625 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:27.017524 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.017494 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-tmp-dir\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.017671 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.017534 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-model-cache\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.017671 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.017583 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5eefe-7410-4de6-b4fe-13488839ec3e-tls-certs\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.017671 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.017619 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-dshm\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.017671 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.017637 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-home\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.017803 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.017705 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtqrl\" (UniqueName: \"kubernetes.io/projected/69d5eefe-7410-4de6-b4fe-13488839ec3e-kube-api-access-dtqrl\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.118699 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.118670 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5eefe-7410-4de6-b4fe-13488839ec3e-tls-certs\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.118699 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.118705 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-dshm\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.118923 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.118724 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-home\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.118923 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.118779 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtqrl\" (UniqueName: \"kubernetes.io/projected/69d5eefe-7410-4de6-b4fe-13488839ec3e-kube-api-access-dtqrl\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.118923 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.118815 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-tmp-dir\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.118923 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.118852 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-model-cache\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.119197 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.119172 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-home\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.119292 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.119201 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-tmp-dir\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.119337 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.119292 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-model-cache\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.121026 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.121003 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-dshm\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.121297 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.121279 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5eefe-7410-4de6-b4fe-13488839ec3e-tls-certs\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.127653 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.127631 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtqrl\" (UniqueName: \"kubernetes.io/projected/69d5eefe-7410-4de6-b4fe-13488839ec3e-kube-api-access-dtqrl\") pod \"e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.225280 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.225196 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:27.368629 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:27.367555 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v"] Jul 02 20:47:27.370901 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:47:27.370873 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69d5eefe_7410_4de6_b4fe_13488839ec3e.slice/crio-42973cba8d3cb5e1019cedd2f445f92f4f2f0c15fcc872e473ce1fb891a2a7f9 WatchSource:0}: Error finding container 42973cba8d3cb5e1019cedd2f445f92f4f2f0c15fcc872e473ce1fb891a2a7f9: Status 404 returned error can't find the container with id 42973cba8d3cb5e1019cedd2f445f92f4f2f0c15fcc872e473ce1fb891a2a7f9 Jul 02 20:47:28.277006 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:28.276972 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" event={"ID":"69d5eefe-7410-4de6-b4fe-13488839ec3e","Type":"ContainerStarted","Data":"f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549"} Jul 02 20:47:28.277006 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:28.277010 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" event={"ID":"69d5eefe-7410-4de6-b4fe-13488839ec3e","Type":"ContainerStarted","Data":"42973cba8d3cb5e1019cedd2f445f92f4f2f0c15fcc872e473ce1fb891a2a7f9"} Jul 02 20:47:28.298381 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:28.298329 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" podStartSLOduration=2.298314353 podStartE2EDuration="2.298314353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:47:28.297488582 +0000 UTC m=+1957.293791815" watchObservedRunningTime="2026-07-02 20:47:28.298314353 +0000 UTC m=+1957.294617639" Jul 02 20:47:30.916925 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:30.916892 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:31.017295 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:31.017263 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:37.225697 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:37.225665 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:37.225697 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:37.225703 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:37.238401 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:37.238380 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:37.332845 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:37.332814 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:47:46.619285 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:46.619175 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:46.951900 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:46.951872 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:52.694075 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:52.694044 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:47:57.684190 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:57.684158 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v"] Jul 02 20:47:57.732167 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:47:57.732132 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:12.705710 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:12.705671 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" podUID="69d5eefe-7410-4de6-b4fe-13488839ec3e" containerName="main" containerID="cri-o://f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549" gracePeriod=45 Jul 02 20:48:12.959094 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:12.959038 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:48:13.037539 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.037506 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5eefe-7410-4de6-b4fe-13488839ec3e-tls-certs\") pod \"69d5eefe-7410-4de6-b4fe-13488839ec3e\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " Jul 02 20:48:13.037719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.037576 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-tmp-dir\") pod \"69d5eefe-7410-4de6-b4fe-13488839ec3e\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " Jul 02 20:48:13.037719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.037603 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtqrl\" (UniqueName: \"kubernetes.io/projected/69d5eefe-7410-4de6-b4fe-13488839ec3e-kube-api-access-dtqrl\") pod \"69d5eefe-7410-4de6-b4fe-13488839ec3e\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " Jul 02 20:48:13.037719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.037623 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-dshm\") pod \"69d5eefe-7410-4de6-b4fe-13488839ec3e\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " Jul 02 20:48:13.037719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.037662 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-home\") pod \"69d5eefe-7410-4de6-b4fe-13488839ec3e\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " Jul 02 20:48:13.037719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.037680 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-model-cache\") pod \"69d5eefe-7410-4de6-b4fe-13488839ec3e\" (UID: \"69d5eefe-7410-4de6-b4fe-13488839ec3e\") " Jul 02 20:48:13.038002 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.037838 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69d5eefe-7410-4de6-b4fe-13488839ec3e" (UID: "69d5eefe-7410-4de6-b4fe-13488839ec3e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:48:13.038002 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.037945 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-home" (OuterVolumeSpecName: "home") pod "69d5eefe-7410-4de6-b4fe-13488839ec3e" (UID: "69d5eefe-7410-4de6-b4fe-13488839ec3e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:48:13.038002 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.037968 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-model-cache" (OuterVolumeSpecName: "model-cache") pod "69d5eefe-7410-4de6-b4fe-13488839ec3e" (UID: "69d5eefe-7410-4de6-b4fe-13488839ec3e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:48:13.038121 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.038003 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-tmp-dir\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:48:13.039857 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.039833 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-dshm" (OuterVolumeSpecName: "dshm") pod "69d5eefe-7410-4de6-b4fe-13488839ec3e" (UID: "69d5eefe-7410-4de6-b4fe-13488839ec3e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:48:13.039967 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.039876 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69d5eefe-7410-4de6-b4fe-13488839ec3e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69d5eefe-7410-4de6-b4fe-13488839ec3e" (UID: "69d5eefe-7410-4de6-b4fe-13488839ec3e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:48:13.039967 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.039886 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d5eefe-7410-4de6-b4fe-13488839ec3e-kube-api-access-dtqrl" (OuterVolumeSpecName: "kube-api-access-dtqrl") pod "69d5eefe-7410-4de6-b4fe-13488839ec3e" (UID: "69d5eefe-7410-4de6-b4fe-13488839ec3e"). InnerVolumeSpecName "kube-api-access-dtqrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:48:13.139466 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.139440 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtqrl\" (UniqueName: \"kubernetes.io/projected/69d5eefe-7410-4de6-b4fe-13488839ec3e-kube-api-access-dtqrl\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:48:13.139466 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.139464 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-dshm\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:48:13.139466 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.139473 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-home\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:48:13.139675 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.139482 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69d5eefe-7410-4de6-b4fe-13488839ec3e-model-cache\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:48:13.139675 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.139492 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69d5eefe-7410-4de6-b4fe-13488839ec3e-tls-certs\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:48:13.490747 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.490706 2565 generic.go:356] "Generic (PLEG): container finished" podID="69d5eefe-7410-4de6-b4fe-13488839ec3e" containerID="f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549" exitCode=0 Jul 02 20:48:13.490747 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.490748 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" Jul 02 20:48:13.491000 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.490761 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" event={"ID":"69d5eefe-7410-4de6-b4fe-13488839ec3e","Type":"ContainerDied","Data":"f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549"} Jul 02 20:48:13.491000 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.490800 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v" event={"ID":"69d5eefe-7410-4de6-b4fe-13488839ec3e","Type":"ContainerDied","Data":"42973cba8d3cb5e1019cedd2f445f92f4f2f0c15fcc872e473ce1fb891a2a7f9"} Jul 02 20:48:13.491000 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.490821 2565 scope.go:117] "RemoveContainer" containerID="f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549" Jul 02 20:48:13.500090 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.500074 2565 scope.go:117] "RemoveContainer" containerID="f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549" Jul 02 20:48:13.500371 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:48:13.500351 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549\": container with ID starting with f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549 not found: ID does not exist" containerID="f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549" Jul 02 20:48:13.500438 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.500379 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549"} err="failed to get container status \"f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549\": rpc error: code = NotFound desc = could not find container \"f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549\": container with ID starting with f9d082b8754b73e0959a76080fac2ed71acb9be676b1402a57308deccf7e8549 not found: ID does not exist" Jul 02 20:48:13.517454 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.517422 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v"] Jul 02 20:48:13.519561 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.519531 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-13937e3a/e2e-lifecycle-model-13937e3a-kserve-7dd4c7bf55-szg5v"] Jul 02 20:48:13.572380 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:13.572345 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69d5eefe-7410-4de6-b4fe-13488839ec3e" path="/var/lib/kubelet/pods/69d5eefe-7410-4de6-b4fe-13488839ec3e/volumes" Jul 02 20:48:33.829552 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:33.829522 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:35.913986 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:35.913950 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:41.542421 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:41.542388 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:41.606614 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:41.606574 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:41.833485 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:41.833404 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:42.079104 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:42.079072 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:43.181314 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:43.181281 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:43.281599 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:43.281574 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:43.381066 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:43.381038 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:44.157150 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:44.157123 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:44.244548 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:44.244521 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:44.343764 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:44.343724 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:45.736163 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:45.736132 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:51.452637 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:51.452600 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:51.543940 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:51.543904 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:52.192080 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:52.192048 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:58.095490 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:58.095457 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:58.198921 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:58.198890 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:48:58.297834 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:48:58.297800 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:08.830719 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:08.830672 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:09.204472 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:09.204437 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:30.180375 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:30.180344 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:36.164427 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:36.164393 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:36.225409 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:36.225377 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:36.523710 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:36.523616 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:36.864397 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:36.864319 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:37.283960 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:37.283929 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:39.546606 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:39.546570 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:39.636670 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:39.636640 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:39.704874 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:39.704815 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:40.006416 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:40.006383 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:40.119175 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:40.119132 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:40.220720 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:40.220675 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:40.778308 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:40.778273 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:40.878616 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:40.878591 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:44.042445 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:44.042388 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:44.372052 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:44.371962 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:44.678096 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:44.678054 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:44.945083 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:44.944971 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:47.636588 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:47.636532 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:47.815815 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:47.815771 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:47.916369 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:47.916340 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:48.300894 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:48.300803 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:48.399284 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:48.399250 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:48.500026 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:48.499996 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:49:51.592694 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:51.592666 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:49:51.598991 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:51.598972 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:49:52.122281 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:49:52.122237 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:50:00.817691 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:00.817657 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:50:00.918902 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:00.918870 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:50:02.398377 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.398331 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6"] Jul 02 20:50:02.398890 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.398871 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69d5eefe-7410-4de6-b4fe-13488839ec3e" containerName="main" Jul 02 20:50:02.398973 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.398895 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="69d5eefe-7410-4de6-b4fe-13488839ec3e" containerName="main" Jul 02 20:50:02.399027 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.399000 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="69d5eefe-7410-4de6-b4fe-13488839ec3e" containerName="main" Jul 02 20:50:02.401939 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.401918 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.408090 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.408068 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-587503\"/\"default-dockercfg-gx4tr\"" Jul 02 20:50:02.408231 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.408116 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-587503\"/\"test-model-587503-kserve-self-signed-certs\"" Jul 02 20:50:02.408381 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.408361 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-587503\"/\"kube-root-ca.crt\"" Jul 02 20:50:02.408476 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.408367 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-587503\"/\"openshift-service-ca.crt\"" Jul 02 20:50:02.413332 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.413308 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6"] Jul 02 20:50:02.501275 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.501242 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:50:02.525423 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.525391 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-home\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.525600 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.525440 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-dshm\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.525600 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.525485 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tmp-dir\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.525600 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.525510 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4hn5\" (UniqueName: \"kubernetes.io/projected/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-kube-api-access-w4hn5\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.525600 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.525529 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-model-cache\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.525600 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.525593 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tls-certs\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.626168 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.626128 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-home\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.626168 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.626176 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-dshm\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.626435 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.626192 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tmp-dir\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.626435 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.626242 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4hn5\" (UniqueName: \"kubernetes.io/projected/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-kube-api-access-w4hn5\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.626435 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.626259 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-model-cache\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.626435 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.626289 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tls-certs\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.626628 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.626580 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-home\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.626682 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.626651 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tmp-dir\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.626721 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.626685 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-model-cache\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.628625 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.628602 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-dshm\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.628788 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.628773 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tls-certs\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.635572 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.635548 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4hn5\" (UniqueName: \"kubernetes.io/projected/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-kube-api-access-w4hn5\") pod \"test-model-587503-kserve-84bd75f9d5-qdmw6\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.711524 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.711443 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:02.847736 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:02.847713 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6"] Jul 02 20:50:03.001561 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:03.001462 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" event={"ID":"febfb4dc-d9c8-4b63-9a4e-1c471b262fab","Type":"ContainerStarted","Data":"d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd"} Jul 02 20:50:03.001561 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:03.001510 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" event={"ID":"febfb4dc-d9c8-4b63-9a4e-1c471b262fab","Type":"ContainerStarted","Data":"ac710855dc48e6da3844b76968cb9e382db9bd04eb2d7f6764c12a017d3d8184"} Jul 02 20:50:03.020973 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:03.020921 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" podStartSLOduration=1.020905366 podStartE2EDuration="1.020905366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:50:03.018553158 +0000 UTC m=+2112.014856393" watchObservedRunningTime="2026-07-02 20:50:03.020905366 +0000 UTC m=+2112.017208653" Jul 02 20:50:09.956073 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:09.956037 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:50:10.056779 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:10.056746 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:50:12.712017 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:12.711973 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:12.712517 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:12.712037 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:12.724470 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:12.724443 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:13.060131 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:13.060037 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:50:44.348702 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:44.348620 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:50:55.166827 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:50:55.166793 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:05.882012 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:05.881981 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6"] Jul 02 20:51:05.907512 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:05.907485 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:07.013129 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:07.013091 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:07.169217 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:07.169183 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:07.558537 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:07.558508 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:07.658286 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:07.658260 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:07.758317 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:07.758289 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:07.824199 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:07.824115 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:12.635351 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:12.635314 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:20.902595 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:20.902533 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" podUID="febfb4dc-d9c8-4b63-9a4e-1c471b262fab" containerName="main" containerID="cri-o://d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd" gracePeriod=45 Jul 02 20:51:21.145100 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.145074 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:51:21.245679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.245580 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4hn5\" (UniqueName: \"kubernetes.io/projected/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-kube-api-access-w4hn5\") pod \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " Jul 02 20:51:21.245679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.245635 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tls-certs\") pod \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " Jul 02 20:51:21.245679 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.245671 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tmp-dir\") pod \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " Jul 02 20:51:21.245953 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.245735 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-model-cache\") pod \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " Jul 02 20:51:21.245953 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.245758 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-home\") pod \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " Jul 02 20:51:21.245953 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.245793 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-dshm\") pod \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\" (UID: \"febfb4dc-d9c8-4b63-9a4e-1c471b262fab\") " Jul 02 20:51:21.245953 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.245928 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "febfb4dc-d9c8-4b63-9a4e-1c471b262fab" (UID: "febfb4dc-d9c8-4b63-9a4e-1c471b262fab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:51:21.246139 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.246070 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-model-cache" (OuterVolumeSpecName: "model-cache") pod "febfb4dc-d9c8-4b63-9a4e-1c471b262fab" (UID: "febfb4dc-d9c8-4b63-9a4e-1c471b262fab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:51:21.246139 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.246089 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-home" (OuterVolumeSpecName: "home") pod "febfb4dc-d9c8-4b63-9a4e-1c471b262fab" (UID: "febfb4dc-d9c8-4b63-9a4e-1c471b262fab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:51:21.246139 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.246098 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tmp-dir\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:51:21.248099 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.248062 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-dshm" (OuterVolumeSpecName: "dshm") pod "febfb4dc-d9c8-4b63-9a4e-1c471b262fab" (UID: "febfb4dc-d9c8-4b63-9a4e-1c471b262fab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:51:21.248226 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.248151 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-kube-api-access-w4hn5" (OuterVolumeSpecName: "kube-api-access-w4hn5") pod "febfb4dc-d9c8-4b63-9a4e-1c471b262fab" (UID: "febfb4dc-d9c8-4b63-9a4e-1c471b262fab"). InnerVolumeSpecName "kube-api-access-w4hn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:51:21.248226 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.248153 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "febfb4dc-d9c8-4b63-9a4e-1c471b262fab" (UID: "febfb4dc-d9c8-4b63-9a4e-1c471b262fab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:51:21.346842 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.346803 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-tls-certs\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:51:21.346842 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.346832 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-model-cache\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:51:21.346842 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.346840 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-home\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:51:21.346842 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.346848 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-dshm\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:51:21.347096 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.346855 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4hn5\" (UniqueName: \"kubernetes.io/projected/febfb4dc-d9c8-4b63-9a4e-1c471b262fab-kube-api-access-w4hn5\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:51:21.377639 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.377607 2565 generic.go:356] "Generic (PLEG): container finished" podID="febfb4dc-d9c8-4b63-9a4e-1c471b262fab" containerID="d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd" exitCode=0 Jul 02 20:51:21.377786 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.377649 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" Jul 02 20:51:21.377786 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.377667 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" event={"ID":"febfb4dc-d9c8-4b63-9a4e-1c471b262fab","Type":"ContainerDied","Data":"d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd"} Jul 02 20:51:21.377786 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.377704 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6" event={"ID":"febfb4dc-d9c8-4b63-9a4e-1c471b262fab","Type":"ContainerDied","Data":"ac710855dc48e6da3844b76968cb9e382db9bd04eb2d7f6764c12a017d3d8184"} Jul 02 20:51:21.377786 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.377719 2565 scope.go:117] "RemoveContainer" containerID="d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd" Jul 02 20:51:21.387004 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.386987 2565 scope.go:117] "RemoveContainer" containerID="d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd" Jul 02 20:51:21.387273 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:51:21.387251 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd\": container with ID starting with d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd not found: ID does not exist" containerID="d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd" Jul 02 20:51:21.387350 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.387281 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd"} err="failed to get container status \"d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd\": rpc error: code = NotFound desc = could not find container \"d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd\": container with ID starting with d4fe6a6596f0958c0404e5c936ae4402cfcbab77d36e4c3753f48072d59192bd not found: ID does not exist" Jul 02 20:51:21.398407 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.398385 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6"] Jul 02 20:51:21.402574 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.402555 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-587503/test-model-587503-kserve-84bd75f9d5-qdmw6"] Jul 02 20:51:21.572936 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:21.572855 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="febfb4dc-d9c8-4b63-9a4e-1c471b262fab" path="/var/lib/kubelet/pods/febfb4dc-d9c8-4b63-9a4e-1c471b262fab/volumes" Jul 02 20:51:31.249397 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:31.249359 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:49.524595 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:49.524557 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:49.789529 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:49.789442 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:50.050638 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:50.050550 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:50.191096 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:50.191060 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:50.587439 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:50.587410 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:50.687770 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:50.687727 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:50.836998 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:50.836972 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:56.573403 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:56.573372 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:56.644946 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:56.644914 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:56.844644 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:56.844553 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:57.077469 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:57.077437 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:57.419022 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:57.418982 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:57.733070 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:57.732992 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:51:57.965439 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:51:57.965413 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:00.642576 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:00.642548 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:00.742929 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:00.742898 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:00.975720 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:00.975690 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:01.075655 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:01.075618 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:01.535751 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:01.535716 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:04.941376 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:04.941346 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:05.315572 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:05.315482 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:05.645902 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:05.645809 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:05.987189 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:05.987159 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:08.395264 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:08.395230 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:08.488119 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:08.488084 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:08.684010 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:08.683967 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:08.867555 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:08.867517 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:09.155112 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:09.155079 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:09.255445 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:09.255415 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:12.634881 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:12.634791 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:12.735939 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:12.735885 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:12.874159 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:12.874128 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:16.758226 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:16.758187 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:16.859482 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:16.859446 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:21.014924 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:21.014870 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:21.364471 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:21.364346 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:21.546300 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:21.546266 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:21.646547 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:21.646461 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:21.857176 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:21.857140 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:27.201159 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:27.201126 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:27.307182 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:27.307149 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:27.456737 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:27.456647 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:27.740732 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:27.740642 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:27.894883 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:27.894848 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:28.354041 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:28.354015 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:28.457089 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:28.457062 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:28.621048 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:28.620973 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:34.439651 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:34.439620 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:34.540367 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:34.540336 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:34.705430 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:34.705336 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:34.932325 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:34.932285 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:35.276505 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:35.276477 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:35.592095 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:35.591987 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:36.031267 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:36.031233 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:38.536952 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:38.536913 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:38.623511 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:38.623480 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:38.696467 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:38.696428 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:39.002860 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:39.002806 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:39.182549 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:39.182519 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:39.308245 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:39.308117 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:43.032876 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:43.032839 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:43.387101 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:43.387008 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:43.692845 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:43.692815 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:44.048121 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:44.048038 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:46.578013 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:46.577961 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:46.677443 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:46.677412 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:46.843357 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:46.843244 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:47.348285 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:47.348250 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:47.464433 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:47.464403 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:47.558315 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:47.558284 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:51.089104 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:51.089071 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:59.865666 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:59.865613 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:52:59.968581 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:52:59.968551 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:53:08.962437 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:08.962401 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:53:09.064352 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:09.064321 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:53:11.904450 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:11.904413 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w"] Jul 02 20:53:11.905441 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:11.905398 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="febfb4dc-d9c8-4b63-9a4e-1c471b262fab" containerName="main" Jul 02 20:53:11.905441 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:11.905442 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="febfb4dc-d9c8-4b63-9a4e-1c471b262fab" containerName="main" Jul 02 20:53:11.905709 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:11.905694 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="febfb4dc-d9c8-4b63-9a4e-1c471b262fab" containerName="main" Jul 02 20:53:11.909557 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:11.909534 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:11.914593 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:11.914560 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bf7336\"/\"kube-root-ca.crt\"" Jul 02 20:53:11.914752 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:11.914681 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bf7336\"/\"openshift-service-ca.crt\"" Jul 02 20:53:11.915161 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:11.915137 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bf7336\"/\"auth-test-model-bf7336-kserve-self-signed-certs\"" Jul 02 20:53:11.915680 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:11.915657 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bf7336\"/\"default-dockercfg-knrj2\"" Jul 02 20:53:11.926579 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:11.926549 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w"] Jul 02 20:53:11.937810 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:11.937788 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:53:12.022636 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.022596 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-model-cache\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.022636 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.022636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-tmp-dir\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.022865 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.022664 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85cf4e25-6da7-4118-b792-d6f206af366d-tls-certs\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.022865 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.022749 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-dshm\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.022865 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.022812 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnr5x\" (UniqueName: \"kubernetes.io/projected/85cf4e25-6da7-4118-b792-d6f206af366d-kube-api-access-fnr5x\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.022971 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.022884 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-home\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.123583 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.123550 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnr5x\" (UniqueName: \"kubernetes.io/projected/85cf4e25-6da7-4118-b792-d6f206af366d-kube-api-access-fnr5x\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.123780 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.123591 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-home\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.123780 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.123647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-model-cache\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.123780 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.123662 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-tmp-dir\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.123780 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.123687 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85cf4e25-6da7-4118-b792-d6f206af366d-tls-certs\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.123780 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.123717 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-dshm\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.124081 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.124057 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-home\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.124081 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.124072 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-tmp-dir\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.124197 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.124115 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-model-cache\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.126054 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.126033 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-dshm\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.126341 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.126317 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85cf4e25-6da7-4118-b792-d6f206af366d-tls-certs\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.133457 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.133433 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnr5x\" (UniqueName: \"kubernetes.io/projected/85cf4e25-6da7-4118-b792-d6f206af366d-kube-api-access-fnr5x\") pod \"auth-test-model-bf7336-kserve-76979c546c-zzw4w\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.221324 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.221252 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:12.354368 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.354344 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w"] Jul 02 20:53:12.356707 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:53:12.356666 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85cf4e25_6da7_4118_b792_d6f206af366d.slice/crio-bc2c518109383e009c6046925c5e3c8ddea7d77b08cd0f09dcbf538a82a37896 WatchSource:0}: Error finding container bc2c518109383e009c6046925c5e3c8ddea7d77b08cd0f09dcbf538a82a37896: Status 404 returned error can't find the container with id bc2c518109383e009c6046925c5e3c8ddea7d77b08cd0f09dcbf538a82a37896 Jul 02 20:53:12.905888 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.905854 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" event={"ID":"85cf4e25-6da7-4118-b792-d6f206af366d","Type":"ContainerStarted","Data":"66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e"} Jul 02 20:53:12.905888 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.905892 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" event={"ID":"85cf4e25-6da7-4118-b792-d6f206af366d","Type":"ContainerStarted","Data":"bc2c518109383e009c6046925c5e3c8ddea7d77b08cd0f09dcbf538a82a37896"} Jul 02 20:53:12.949376 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:12.949328 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" podStartSLOduration=1.9493127399999999 podStartE2EDuration="1.94931274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:53:12.948868568 +0000 UTC m=+2301.945171801" watchObservedRunningTime="2026-07-02 20:53:12.94931274 +0000 UTC m=+2301.945616019" Jul 02 20:53:22.221772 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:22.221738 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:22.222323 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:22.221796 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:22.234046 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:22.234021 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:22.963768 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:22.963738 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:53:30.641121 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:30.641091 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:53:48.448219 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:53:48.448183 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:54:09.900550 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:54:09.900517 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:54:51.621066 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:54:51.621042 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:54:51.635092 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:54:51.635067 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:55:02.984570 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:02.984533 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:55:13.314529 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:13.314448 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:55:13.377541 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:13.377503 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:55:27.637192 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:27.637147 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w"] Jul 02 20:55:27.658504 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:27.658475 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:55:28.625571 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:28.625536 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:55:33.303048 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.303007 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq"] Jul 02 20:55:33.307828 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.307803 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.310848 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.310822 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bf7336\"/\"sub-test-model-bf7336-kserve-self-signed-certs\"" Jul 02 20:55:33.317375 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.317353 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq"] Jul 02 20:55:33.333041 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.333011 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:55:33.416359 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.416324 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tmp-dir\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.416359 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.416361 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2zrb\" (UniqueName: \"kubernetes.io/projected/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-kube-api-access-z2zrb\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.416581 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.416430 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-dshm\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.416581 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.416447 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-model-cache\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.416581 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.416464 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-home\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.416581 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.416538 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tls-certs\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.517658 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.517623 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tmp-dir\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.517658 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.517661 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2zrb\" (UniqueName: \"kubernetes.io/projected/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-kube-api-access-z2zrb\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.517947 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.517706 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-dshm\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.517947 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.517721 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-model-cache\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.517947 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.517737 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-home\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.517947 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.517767 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tls-certs\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.518159 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.518104 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tmp-dir\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.518201 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.518159 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-model-cache\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.518281 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.518193 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-home\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.520020 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.520003 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-dshm\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.520414 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.520395 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tls-certs\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.525825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.525802 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2zrb\" (UniqueName: \"kubernetes.io/projected/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-kube-api-access-z2zrb\") pod \"sub-test-model-bf7336-kserve-747c49f86f-xfzhq\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.619706 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.619618 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:33.754660 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:33.754607 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq"] Jul 02 20:55:33.757159 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:55:33.757133 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4f3d1580_c141_4e2f_b98c_4f6b80280fc1.slice/crio-0e2a62afb40cb12a241505e5a2c9019ec0a7924ae4d804b2e5e4c3cdd2f6af1c WatchSource:0}: Error finding container 0e2a62afb40cb12a241505e5a2c9019ec0a7924ae4d804b2e5e4c3cdd2f6af1c: Status 404 returned error can't find the container with id 0e2a62afb40cb12a241505e5a2c9019ec0a7924ae4d804b2e5e4c3cdd2f6af1c Jul 02 20:55:34.574739 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:34.574698 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" event={"ID":"4f3d1580-c141-4e2f-b98c-4f6b80280fc1","Type":"ContainerStarted","Data":"e4a45acc0b9b6bbb58c8eaf57139c13a4986ccf694adec5ccffb722c76eecb68"} Jul 02 20:55:34.574739 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:34.574737 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" event={"ID":"4f3d1580-c141-4e2f-b98c-4f6b80280fc1","Type":"ContainerStarted","Data":"0e2a62afb40cb12a241505e5a2c9019ec0a7924ae4d804b2e5e4c3cdd2f6af1c"} Jul 02 20:55:34.592312 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:34.592268 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" podStartSLOduration=1.592250746 podStartE2EDuration="1.592250746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:55:34.590659122 +0000 UTC m=+2443.586962354" watchObservedRunningTime="2026-07-02 20:55:34.592250746 +0000 UTC m=+2443.588553978" Jul 02 20:55:42.657823 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:42.657757 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" podUID="85cf4e25-6da7-4118-b792-d6f206af366d" containerName="main" containerID="cri-o://66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e" gracePeriod=45 Jul 02 20:55:42.909748 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:42.909674 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:55:43.004810 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.004774 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-home\") pod \"85cf4e25-6da7-4118-b792-d6f206af366d\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " Jul 02 20:55:43.005003 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.004819 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-tmp-dir\") pod \"85cf4e25-6da7-4118-b792-d6f206af366d\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " Jul 02 20:55:43.005003 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.004885 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnr5x\" (UniqueName: \"kubernetes.io/projected/85cf4e25-6da7-4118-b792-d6f206af366d-kube-api-access-fnr5x\") pod \"85cf4e25-6da7-4118-b792-d6f206af366d\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " Jul 02 20:55:43.005003 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.004904 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-model-cache\") pod \"85cf4e25-6da7-4118-b792-d6f206af366d\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " Jul 02 20:55:43.005003 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.004927 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85cf4e25-6da7-4118-b792-d6f206af366d-tls-certs\") pod \"85cf4e25-6da7-4118-b792-d6f206af366d\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " Jul 02 20:55:43.005003 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.004989 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-dshm\") pod \"85cf4e25-6da7-4118-b792-d6f206af366d\" (UID: \"85cf4e25-6da7-4118-b792-d6f206af366d\") " Jul 02 20:55:43.005313 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.005073 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-home" (OuterVolumeSpecName: "home") pod "85cf4e25-6da7-4118-b792-d6f206af366d" (UID: "85cf4e25-6da7-4118-b792-d6f206af366d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:55:43.005313 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.005163 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "85cf4e25-6da7-4118-b792-d6f206af366d" (UID: "85cf4e25-6da7-4118-b792-d6f206af366d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:55:43.005313 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.005175 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-model-cache" (OuterVolumeSpecName: "model-cache") pod "85cf4e25-6da7-4118-b792-d6f206af366d" (UID: "85cf4e25-6da7-4118-b792-d6f206af366d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:55:43.005462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.005403 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-model-cache\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:55:43.005462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.005423 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-home\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:55:43.005462 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.005437 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-tmp-dir\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:55:43.007294 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.007271 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85cf4e25-6da7-4118-b792-d6f206af366d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "85cf4e25-6da7-4118-b792-d6f206af366d" (UID: "85cf4e25-6da7-4118-b792-d6f206af366d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:55:43.007402 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.007307 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-dshm" (OuterVolumeSpecName: "dshm") pod "85cf4e25-6da7-4118-b792-d6f206af366d" (UID: "85cf4e25-6da7-4118-b792-d6f206af366d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:55:43.007402 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.007323 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85cf4e25-6da7-4118-b792-d6f206af366d-kube-api-access-fnr5x" (OuterVolumeSpecName: "kube-api-access-fnr5x") pod "85cf4e25-6da7-4118-b792-d6f206af366d" (UID: "85cf4e25-6da7-4118-b792-d6f206af366d"). InnerVolumeSpecName "kube-api-access-fnr5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:55:43.105922 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.105888 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnr5x\" (UniqueName: \"kubernetes.io/projected/85cf4e25-6da7-4118-b792-d6f206af366d-kube-api-access-fnr5x\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:55:43.105922 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.105918 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85cf4e25-6da7-4118-b792-d6f206af366d-tls-certs\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:55:43.105922 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.105929 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85cf4e25-6da7-4118-b792-d6f206af366d-dshm\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:55:43.620497 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.620471 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:43.620685 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.620522 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:43.622949 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.622922 2565 generic.go:356] "Generic (PLEG): container finished" podID="85cf4e25-6da7-4118-b792-d6f206af366d" containerID="66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e" exitCode=0 Jul 02 20:55:43.623082 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.622960 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" Jul 02 20:55:43.623082 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.623006 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" event={"ID":"85cf4e25-6da7-4118-b792-d6f206af366d","Type":"ContainerDied","Data":"66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e"} Jul 02 20:55:43.623082 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.623063 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w" event={"ID":"85cf4e25-6da7-4118-b792-d6f206af366d","Type":"ContainerDied","Data":"bc2c518109383e009c6046925c5e3c8ddea7d77b08cd0f09dcbf538a82a37896"} Jul 02 20:55:43.623266 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.623085 2565 scope.go:117] "RemoveContainer" containerID="66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e" Jul 02 20:55:43.631869 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.631852 2565 scope.go:117] "RemoveContainer" containerID="66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e" Jul 02 20:55:43.632137 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:55:43.632117 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e\": container with ID starting with 66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e not found: ID does not exist" containerID="66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e" Jul 02 20:55:43.632197 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.632146 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e"} err="failed to get container status \"66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e\": rpc error: code = NotFound desc = could not find container \"66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e\": container with ID starting with 66ef177960f015efcd086880ba1d2b9dae1c525f3c1b725cb4112d0f95f0104e not found: ID does not exist" Jul 02 20:55:43.634575 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.634558 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:43.642828 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.642807 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w"] Jul 02 20:55:43.649379 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:43.649360 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/auth-test-model-bf7336-kserve-76979c546c-zzw4w"] Jul 02 20:55:44.640761 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:44.640726 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:55:45.571673 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:55:45.571643 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85cf4e25-6da7-4118-b792-d6f206af366d" path="/var/lib/kubelet/pods/85cf4e25-6da7-4118-b792-d6f206af366d/volumes" Jul 02 20:56:02.605421 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.605388 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr"] Jul 02 20:56:02.606020 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.606002 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85cf4e25-6da7-4118-b792-d6f206af366d" containerName="main" Jul 02 20:56:02.606070 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.606025 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="85cf4e25-6da7-4118-b792-d6f206af366d" containerName="main" Jul 02 20:56:02.606139 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.606128 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="85cf4e25-6da7-4118-b792-d6f206af366d" containerName="main" Jul 02 20:56:02.609060 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.609027 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.612002 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.611977 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"default-dockercfg-wr24g\"" Jul 02 20:56:02.612002 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.611995 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"kube-root-ca.crt\"" Jul 02 20:56:02.612185 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.612010 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"openshift-service-ca.crt\"" Jul 02 20:56:02.613424 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.613407 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-9994a6\"/\"sub-test-model-9994a6-kserve-self-signed-certs\"" Jul 02 20:56:02.617969 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.617945 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr"] Jul 02 20:56:02.641013 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.640985 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:56:02.673400 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.673374 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tls-certs\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.673518 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.673411 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-model-cache\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.673518 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.673499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-dshm\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.673645 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.673538 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-home\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.673645 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.673572 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzvnm\" (UniqueName: \"kubernetes.io/projected/2b43d65b-9577-4671-aefd-5cbb6851c0f8-kube-api-access-rzvnm\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.673645 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.673622 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tmp-dir\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.775059 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.775023 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-dshm\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.775269 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.775077 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-home\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.775269 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.775107 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzvnm\" (UniqueName: \"kubernetes.io/projected/2b43d65b-9577-4671-aefd-5cbb6851c0f8-kube-api-access-rzvnm\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.775269 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.775156 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tmp-dir\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.775269 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.775233 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tls-certs\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.775269 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.775265 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-model-cache\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.775592 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.775570 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-home\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.775664 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.775623 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tmp-dir\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.775664 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.775652 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-model-cache\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.777499 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.777471 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-dshm\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.777919 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.777897 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tls-certs\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.783066 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.783048 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzvnm\" (UniqueName: \"kubernetes.io/projected/2b43d65b-9577-4671-aefd-5cbb6851c0f8-kube-api-access-rzvnm\") pod \"sub-test-model-9994a6-kserve-6ff48764c5-dvrcr\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:02.919935 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:02.919907 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:03.053546 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:03.053518 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr"] Jul 02 20:56:03.055156 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:56:03.055135 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b43d65b_9577_4671_aefd_5cbb6851c0f8.slice/crio-f284c6365be29c58dde800368a385fbff1ecc1e7a414d5ed2d5153e32aa73d96 WatchSource:0}: Error finding container f284c6365be29c58dde800368a385fbff1ecc1e7a414d5ed2d5153e32aa73d96: Status 404 returned error can't find the container with id f284c6365be29c58dde800368a385fbff1ecc1e7a414d5ed2d5153e32aa73d96 Jul 02 20:56:03.721313 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:03.721276 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" event={"ID":"2b43d65b-9577-4671-aefd-5cbb6851c0f8","Type":"ContainerStarted","Data":"d5d6c49cd515d554b70cb3624b99c93958bbcf1fc927c883804003043d8942e3"} Jul 02 20:56:03.721313 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:03.721314 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" event={"ID":"2b43d65b-9577-4671-aefd-5cbb6851c0f8","Type":"ContainerStarted","Data":"f284c6365be29c58dde800368a385fbff1ecc1e7a414d5ed2d5153e32aa73d96"} Jul 02 20:56:03.739156 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:03.739115 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" podStartSLOduration=1.739102701 podStartE2EDuration="1.739102701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:56:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:56:03.738447688 +0000 UTC m=+2472.734750921" watchObservedRunningTime="2026-07-02 20:56:03.739102701 +0000 UTC m=+2472.735405933" Jul 02 20:56:12.920836 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:12.920805 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:12.921248 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:12.920852 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:12.933464 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:12.933443 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:13.782036 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:13.782009 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:56:52.784138 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:52.784055 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq"] Jul 02 20:56:52.806851 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:52.806826 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:56:53.815228 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:53.815184 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr"] Jul 02 20:56:53.836375 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:56:53.836342 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:57:07.804397 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:07.804336 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" podUID="4f3d1580-c141-4e2f-b98c-4f6b80280fc1" containerName="main" containerID="cri-o://e4a45acc0b9b6bbb58c8eaf57139c13a4986ccf694adec5ccffb722c76eecb68" gracePeriod=45 Jul 02 20:57:08.032884 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.032846 2565 generic.go:356] "Generic (PLEG): container finished" podID="4f3d1580-c141-4e2f-b98c-4f6b80280fc1" containerID="e4a45acc0b9b6bbb58c8eaf57139c13a4986ccf694adec5ccffb722c76eecb68" exitCode=0 Jul 02 20:57:08.033038 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.032904 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" event={"ID":"4f3d1580-c141-4e2f-b98c-4f6b80280fc1","Type":"ContainerDied","Data":"e4a45acc0b9b6bbb58c8eaf57139c13a4986ccf694adec5ccffb722c76eecb68"} Jul 02 20:57:08.056694 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.056640 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:57:08.196479 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.196448 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-home\") pod \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " Jul 02 20:57:08.196656 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.196501 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-model-cache\") pod \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " Jul 02 20:57:08.196656 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.196573 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-dshm\") pod \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " Jul 02 20:57:08.196656 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.196611 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tmp-dir\") pod \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " Jul 02 20:57:08.196656 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.196652 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tls-certs\") pod \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " Jul 02 20:57:08.196847 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.196693 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2zrb\" (UniqueName: \"kubernetes.io/projected/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-kube-api-access-z2zrb\") pod \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\" (UID: \"4f3d1580-c141-4e2f-b98c-4f6b80280fc1\") " Jul 02 20:57:08.196847 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.196788 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-home" (OuterVolumeSpecName: "home") pod "4f3d1580-c141-4e2f-b98c-4f6b80280fc1" (UID: "4f3d1580-c141-4e2f-b98c-4f6b80280fc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:57:08.196952 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.196857 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-model-cache" (OuterVolumeSpecName: "model-cache") pod "4f3d1580-c141-4e2f-b98c-4f6b80280fc1" (UID: "4f3d1580-c141-4e2f-b98c-4f6b80280fc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:57:08.197010 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.196976 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4f3d1580-c141-4e2f-b98c-4f6b80280fc1" (UID: "4f3d1580-c141-4e2f-b98c-4f6b80280fc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:57:08.197069 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.197057 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tmp-dir\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:08.197117 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.197072 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-home\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:08.197117 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.197084 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-model-cache\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:08.198761 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.198738 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-dshm" (OuterVolumeSpecName: "dshm") pod "4f3d1580-c141-4e2f-b98c-4f6b80280fc1" (UID: "4f3d1580-c141-4e2f-b98c-4f6b80280fc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:57:08.198946 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.198922 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-kube-api-access-z2zrb" (OuterVolumeSpecName: "kube-api-access-z2zrb") pod "4f3d1580-c141-4e2f-b98c-4f6b80280fc1" (UID: "4f3d1580-c141-4e2f-b98c-4f6b80280fc1"). InnerVolumeSpecName "kube-api-access-z2zrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:57:08.199039 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.199018 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4f3d1580-c141-4e2f-b98c-4f6b80280fc1" (UID: "4f3d1580-c141-4e2f-b98c-4f6b80280fc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:57:08.297835 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.297796 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-dshm\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:08.297835 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.297827 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-tls-certs\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:08.297835 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.297839 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2zrb\" (UniqueName: \"kubernetes.io/projected/4f3d1580-c141-4e2f-b98c-4f6b80280fc1-kube-api-access-z2zrb\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:08.835986 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:08.835925 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" podUID="2b43d65b-9577-4671-aefd-5cbb6851c0f8" containerName="main" containerID="cri-o://d5d6c49cd515d554b70cb3624b99c93958bbcf1fc927c883804003043d8942e3" gracePeriod=45 Jul 02 20:57:09.039016 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.038986 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" Jul 02 20:57:09.039016 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.039001 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq" event={"ID":"4f3d1580-c141-4e2f-b98c-4f6b80280fc1","Type":"ContainerDied","Data":"0e2a62afb40cb12a241505e5a2c9019ec0a7924ae4d804b2e5e4c3cdd2f6af1c"} Jul 02 20:57:09.039241 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.039054 2565 scope.go:117] "RemoveContainer" containerID="e4a45acc0b9b6bbb58c8eaf57139c13a4986ccf694adec5ccffb722c76eecb68" Jul 02 20:57:09.040907 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.040881 2565 generic.go:356] "Generic (PLEG): container finished" podID="2b43d65b-9577-4671-aefd-5cbb6851c0f8" containerID="d5d6c49cd515d554b70cb3624b99c93958bbcf1fc927c883804003043d8942e3" exitCode=0 Jul 02 20:57:09.041066 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.040951 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" event={"ID":"2b43d65b-9577-4671-aefd-5cbb6851c0f8","Type":"ContainerDied","Data":"d5d6c49cd515d554b70cb3624b99c93958bbcf1fc927c883804003043d8942e3"} Jul 02 20:57:09.064006 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.063973 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq"] Jul 02 20:57:09.067189 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.067160 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/sub-test-model-bf7336-kserve-747c49f86f-xfzhq"] Jul 02 20:57:09.100387 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.100334 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:57:09.206646 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.206607 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rzvnm\" (UniqueName: \"kubernetes.io/projected/2b43d65b-9577-4671-aefd-5cbb6851c0f8-kube-api-access-rzvnm\") pod \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " Jul 02 20:57:09.206840 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.206653 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-model-cache\") pod \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " Jul 02 20:57:09.206840 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.206683 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tls-certs\") pod \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " Jul 02 20:57:09.206840 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.206735 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-home\") pod \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " Jul 02 20:57:09.206840 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.206828 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tmp-dir\") pod \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " Jul 02 20:57:09.207049 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.206860 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-dshm\") pod \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\" (UID: \"2b43d65b-9577-4671-aefd-5cbb6851c0f8\") " Jul 02 20:57:09.207049 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.206956 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-model-cache" (OuterVolumeSpecName: "model-cache") pod "2b43d65b-9577-4671-aefd-5cbb6851c0f8" (UID: "2b43d65b-9577-4671-aefd-5cbb6851c0f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:57:09.207049 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.207005 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-home" (OuterVolumeSpecName: "home") pod "2b43d65b-9577-4671-aefd-5cbb6851c0f8" (UID: "2b43d65b-9577-4671-aefd-5cbb6851c0f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:57:09.207168 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.207094 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2b43d65b-9577-4671-aefd-5cbb6851c0f8" (UID: "2b43d65b-9577-4671-aefd-5cbb6851c0f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:57:09.207309 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.207293 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-model-cache\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:09.207367 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.207312 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-home\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:09.207367 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.207321 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tmp-dir\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:09.209135 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.209117 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-dshm" (OuterVolumeSpecName: "dshm") pod "2b43d65b-9577-4671-aefd-5cbb6851c0f8" (UID: "2b43d65b-9577-4671-aefd-5cbb6851c0f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:57:09.209363 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.209335 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2b43d65b-9577-4671-aefd-5cbb6851c0f8" (UID: "2b43d65b-9577-4671-aefd-5cbb6851c0f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:57:09.209444 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.209393 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b43d65b-9577-4671-aefd-5cbb6851c0f8-kube-api-access-rzvnm" (OuterVolumeSpecName: "kube-api-access-rzvnm") pod "2b43d65b-9577-4671-aefd-5cbb6851c0f8" (UID: "2b43d65b-9577-4671-aefd-5cbb6851c0f8"). InnerVolumeSpecName "kube-api-access-rzvnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:57:09.308072 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.308043 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b43d65b-9577-4671-aefd-5cbb6851c0f8-dshm\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:09.308072 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.308069 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzvnm\" (UniqueName: \"kubernetes.io/projected/2b43d65b-9577-4671-aefd-5cbb6851c0f8-kube-api-access-rzvnm\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:09.308284 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.308080 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b43d65b-9577-4671-aefd-5cbb6851c0f8-tls-certs\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:57:09.571302 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:09.571270 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f3d1580-c141-4e2f-b98c-4f6b80280fc1" path="/var/lib/kubelet/pods/4f3d1580-c141-4e2f-b98c-4f6b80280fc1/volumes" Jul 02 20:57:10.048475 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:10.048450 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" Jul 02 20:57:10.048904 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:10.048480 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr" event={"ID":"2b43d65b-9577-4671-aefd-5cbb6851c0f8","Type":"ContainerDied","Data":"f284c6365be29c58dde800368a385fbff1ecc1e7a414d5ed2d5153e32aa73d96"} Jul 02 20:57:10.048904 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:10.048528 2565 scope.go:117] "RemoveContainer" containerID="d5d6c49cd515d554b70cb3624b99c93958bbcf1fc927c883804003043d8942e3" Jul 02 20:57:10.067330 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:10.067300 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr"] Jul 02 20:57:10.070645 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:10.070622 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-9994a6/sub-test-model-9994a6-kserve-6ff48764c5-dvrcr"] Jul 02 20:57:11.572130 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:11.572099 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b43d65b-9577-4671-aefd-5cbb6851c0f8" path="/var/lib/kubelet/pods/2b43d65b-9577-4671-aefd-5cbb6851c0f8/volumes" Jul 02 20:57:13.372091 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:13.372042 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:57:13.469560 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:13.469493 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:57:22.006135 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.006100 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt"] Jul 02 20:57:22.006676 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.006655 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b43d65b-9577-4671-aefd-5cbb6851c0f8" containerName="main" Jul 02 20:57:22.006764 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.006680 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b43d65b-9577-4671-aefd-5cbb6851c0f8" containerName="main" Jul 02 20:57:22.006764 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.006703 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f3d1580-c141-4e2f-b98c-4f6b80280fc1" containerName="main" Jul 02 20:57:22.006764 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.006713 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f3d1580-c141-4e2f-b98c-4f6b80280fc1" containerName="main" Jul 02 20:57:22.006921 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.006800 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f3d1580-c141-4e2f-b98c-4f6b80280fc1" containerName="main" Jul 02 20:57:22.006921 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.006818 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b43d65b-9577-4671-aefd-5cbb6851c0f8" containerName="main" Jul 02 20:57:22.011233 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.011191 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.013993 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.013974 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bf7336\"/\"default-dockercfg-knrj2\"" Jul 02 20:57:22.014254 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.014238 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bf7336\"/\"openshift-service-ca.crt\"" Jul 02 20:57:22.015401 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.015383 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bf7336\"/\"kube-root-ca.crt\"" Jul 02 20:57:22.015493 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.015429 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bf7336\"/\"rate-test-model-bf7336-kserve-self-signed-certs\"" Jul 02 20:57:22.019351 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.018854 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt"] Jul 02 20:57:22.024087 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.024063 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tls-certs\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.024190 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.024142 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tmp-dir\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.024190 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.024173 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-home\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.024295 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.024262 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzwnd\" (UniqueName: \"kubernetes.io/projected/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-kube-api-access-zzwnd\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.024345 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.024329 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-dshm\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.024382 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.024354 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-model-cache\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.043427 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.043381 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:57:22.125083 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.125054 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tls-certs\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.125257 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.125105 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tmp-dir\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.125305 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.125257 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-home\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.125345 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.125325 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzwnd\" (UniqueName: \"kubernetes.io/projected/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-kube-api-access-zzwnd\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.125399 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.125382 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-dshm\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.125441 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.125419 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-model-cache\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.125487 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.125457 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tmp-dir\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.125598 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.125571 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-home\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.125735 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.125695 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-model-cache\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.127636 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.127615 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-dshm\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.127836 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.127820 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tls-certs\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.142253 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.142222 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzwnd\" (UniqueName: \"kubernetes.io/projected/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-kube-api-access-zzwnd\") pod \"rate-test-model-bf7336-kserve-895458579-5bdqt\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.322506 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.322425 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:22.456501 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:22.456474 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt"] Jul 02 20:57:22.458829 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:57:22.458804 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48b206b4_fbe9_4102_96c3_52cd2fc8eb01.slice/crio-c77fc57329f59d4abbceacffa8b50de4fdbb1be6c5e5acdfcfec8a3fd2916632 WatchSource:0}: Error finding container c77fc57329f59d4abbceacffa8b50de4fdbb1be6c5e5acdfcfec8a3fd2916632: Status 404 returned error can't find the container with id c77fc57329f59d4abbceacffa8b50de4fdbb1be6c5e5acdfcfec8a3fd2916632 Jul 02 20:57:23.113111 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:23.113075 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" event={"ID":"48b206b4-fbe9-4102-96c3-52cd2fc8eb01","Type":"ContainerStarted","Data":"2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51"} Jul 02 20:57:23.113507 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:23.113122 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" event={"ID":"48b206b4-fbe9-4102-96c3-52cd2fc8eb01","Type":"ContainerStarted","Data":"c77fc57329f59d4abbceacffa8b50de4fdbb1be6c5e5acdfcfec8a3fd2916632"} Jul 02 20:57:23.131802 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:23.131752 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" podStartSLOduration=2.131738926 podStartE2EDuration="2.131738926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 20:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:57:23.130018191 +0000 UTC m=+2552.126321426" watchObservedRunningTime="2026-07-02 20:57:23.131738926 +0000 UTC m=+2552.128042159" Jul 02 20:57:32.323170 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:32.323126 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:32.323170 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:32.323179 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:32.336257 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:32.336200 2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:33.173072 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:33.173035 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:57:43.643463 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:57:43.643425 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:00.572422 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:00.572388 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:00.672142 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:00.672110 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:09.831949 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:09.831908 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:09.931198 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:09.931169 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:19.902609 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:19.902528 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:27.796028 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:27.795998 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:27.861579 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:27.861546 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:29.808545 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:29.808516 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt"] Jul 02 20:58:29.826809 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:29.826780 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:30.722408 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:30.722378 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:31.656389 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:31.656350 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6908-140273651941984"] Jul 02 20:58:31.659723 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:31.659701 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651941984" Jul 02 20:58:31.666091 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:31.666043 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6908-140273651941984"] Jul 02 20:58:31.759674 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:31.759636 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvk8\" (UniqueName: \"kubernetes.io/projected/a2365ce9-812d-41f8-946b-f49548630e47-kube-api-access-6kvk8\") pod \"test-curl-6908-140273651941984\" (UID: \"a2365ce9-812d-41f8-946b-f49548630e47\") " pod="opendatahub/test-curl-6908-140273651941984" Jul 02 20:58:31.860636 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:31.860602 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvk8\" (UniqueName: \"kubernetes.io/projected/a2365ce9-812d-41f8-946b-f49548630e47-kube-api-access-6kvk8\") pod \"test-curl-6908-140273651941984\" (UID: \"a2365ce9-812d-41f8-946b-f49548630e47\") " pod="opendatahub/test-curl-6908-140273651941984" Jul 02 20:58:31.869193 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:31.869166 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvk8\" (UniqueName: \"kubernetes.io/projected/a2365ce9-812d-41f8-946b-f49548630e47-kube-api-access-6kvk8\") pod \"test-curl-6908-140273651941984\" (UID: \"a2365ce9-812d-41f8-946b-f49548630e47\") " pod="opendatahub/test-curl-6908-140273651941984" Jul 02 20:58:31.970824 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:31.970743 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651941984" Jul 02 20:58:32.098858 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:32.098833 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6908-140273651941984"] Jul 02 20:58:32.101028 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:58:32.101006 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2365ce9_812d_41f8_946b_f49548630e47.slice/crio-df1e2f757083297cf6e15a719f26e699fa40b325a6cb880347f3a6a1b3a9909a WatchSource:0}: Error finding container df1e2f757083297cf6e15a719f26e699fa40b325a6cb880347f3a6a1b3a9909a: Status 404 returned error can't find the container with id df1e2f757083297cf6e15a719f26e699fa40b325a6cb880347f3a6a1b3a9909a Jul 02 20:58:32.102727 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:32.102708 2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 20:58:32.446069 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:32.446000 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6908-140273651941984" event={"ID":"a2365ce9-812d-41f8-946b-f49548630e47","Type":"ContainerStarted","Data":"df1e2f757083297cf6e15a719f26e699fa40b325a6cb880347f3a6a1b3a9909a"} Jul 02 20:58:33.451814 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:33.451775 2565 generic.go:356] "Generic (PLEG): container finished" podID="a2365ce9-812d-41f8-946b-f49548630e47" containerID="c277b587f7880a9d325cedb0ebd31199ab0bc2699abdc592fadd8b3193496a2e" exitCode=0 Jul 02 20:58:33.452182 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:33.451870 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6908-140273651941984" event={"ID":"a2365ce9-812d-41f8-946b-f49548630e47","Type":"ContainerDied","Data":"c277b587f7880a9d325cedb0ebd31199ab0bc2699abdc592fadd8b3193496a2e"} Jul 02 20:58:34.590871 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.590843 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651941984" Jul 02 20:58:34.609519 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.609494 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6908-140273651941984_a2365ce9-812d-41f8-946b-f49548630e47/test-curl-6908-140273651941984/0.log" Jul 02 20:58:34.635607 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.635578 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6908-140273651941984"] Jul 02 20:58:34.638345 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.638321 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6908-140273651941984"] Jul 02 20:58:34.688706 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.688678 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kvk8\" (UniqueName: \"kubernetes.io/projected/a2365ce9-812d-41f8-946b-f49548630e47-kube-api-access-6kvk8\") pod \"a2365ce9-812d-41f8-946b-f49548630e47\" (UID: \"a2365ce9-812d-41f8-946b-f49548630e47\") " Jul 02 20:58:34.690896 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.690873 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2365ce9-812d-41f8-946b-f49548630e47-kube-api-access-6kvk8" (OuterVolumeSpecName: "kube-api-access-6kvk8") pod "a2365ce9-812d-41f8-946b-f49548630e47" (UID: "a2365ce9-812d-41f8-946b-f49548630e47"). InnerVolumeSpecName "kube-api-access-6kvk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:34.765749 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.765680 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6908-140273651942096"] Jul 02 20:58:34.766049 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.766037 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2365ce9-812d-41f8-946b-f49548630e47" containerName="test-curl-6908-140273651941984" Jul 02 20:58:34.766103 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.766051 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2365ce9-812d-41f8-946b-f49548630e47" containerName="test-curl-6908-140273651941984" Jul 02 20:58:34.766140 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.766127 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2365ce9-812d-41f8-946b-f49548630e47" containerName="test-curl-6908-140273651941984" Jul 02 20:58:34.768542 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.768527 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651942096" Jul 02 20:58:34.775626 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.775606 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6908-140273651942096"] Jul 02 20:58:34.789673 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.789653 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvk8\" (UniqueName: \"kubernetes.io/projected/a2365ce9-812d-41f8-946b-f49548630e47-kube-api-access-6kvk8\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:34.890445 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.890406 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lg7n\" (UniqueName: \"kubernetes.io/projected/2a3717a1-6204-4b52-a74e-ae5dfb00e9b3-kube-api-access-9lg7n\") pod \"test-curl-6908-140273651942096\" (UID: \"2a3717a1-6204-4b52-a74e-ae5dfb00e9b3\") " pod="opendatahub/test-curl-6908-140273651942096" Jul 02 20:58:34.991629 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.991595 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lg7n\" (UniqueName: \"kubernetes.io/projected/2a3717a1-6204-4b52-a74e-ae5dfb00e9b3-kube-api-access-9lg7n\") pod \"test-curl-6908-140273651942096\" (UID: \"2a3717a1-6204-4b52-a74e-ae5dfb00e9b3\") " pod="opendatahub/test-curl-6908-140273651942096" Jul 02 20:58:34.999818 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:34.999791 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lg7n\" (UniqueName: \"kubernetes.io/projected/2a3717a1-6204-4b52-a74e-ae5dfb00e9b3-kube-api-access-9lg7n\") pod \"test-curl-6908-140273651942096\" (UID: \"2a3717a1-6204-4b52-a74e-ae5dfb00e9b3\") " pod="opendatahub/test-curl-6908-140273651942096" Jul 02 20:58:35.077652 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:35.077560 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651942096" Jul 02 20:58:35.211354 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:35.211329 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6908-140273651942096"] Jul 02 20:58:35.213054 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:58:35.213025 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a3717a1_6204_4b52_a74e_ae5dfb00e9b3.slice/crio-dd641f9186c2e4a38bae75f496fbcb7c5c2f76bde56ac0a7c66d201b86c36af6 WatchSource:0}: Error finding container dd641f9186c2e4a38bae75f496fbcb7c5c2f76bde56ac0a7c66d201b86c36af6: Status 404 returned error can't find the container with id dd641f9186c2e4a38bae75f496fbcb7c5c2f76bde56ac0a7c66d201b86c36af6 Jul 02 20:58:35.462731 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:35.462697 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6908-140273651942096" event={"ID":"2a3717a1-6204-4b52-a74e-ae5dfb00e9b3","Type":"ContainerStarted","Data":"dd641f9186c2e4a38bae75f496fbcb7c5c2f76bde56ac0a7c66d201b86c36af6"} Jul 02 20:58:35.463790 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:35.463773 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df1e2f757083297cf6e15a719f26e699fa40b325a6cb880347f3a6a1b3a9909a" Jul 02 20:58:35.463893 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:35.463803 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651941984" Jul 02 20:58:35.477441 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:35.477414 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273651941984\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="a2365ce9-812d-41f8-946b-f49548630e47" pod="opendatahub/test-curl-6908-140273651941984" Jul 02 20:58:35.572389 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:35.572360 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2365ce9-812d-41f8-946b-f49548630e47" path="/var/lib/kubelet/pods/a2365ce9-812d-41f8-946b-f49548630e47/volumes" Jul 02 20:58:36.470317 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:36.470280 2565 generic.go:356] "Generic (PLEG): container finished" podID="2a3717a1-6204-4b52-a74e-ae5dfb00e9b3" containerID="84dca8e41c2f86d0a7560ed8e720a5ffc86e977244aee900848fedebac171cd9" exitCode=0 Jul 02 20:58:36.470695 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:36.470361 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6908-140273651942096" event={"ID":"2a3717a1-6204-4b52-a74e-ae5dfb00e9b3","Type":"ContainerDied","Data":"84dca8e41c2f86d0a7560ed8e720a5ffc86e977244aee900848fedebac171cd9"} Jul 02 20:58:37.602716 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.602688 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651942096" Jul 02 20:58:37.623029 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.623006 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6908-140273651942096_2a3717a1-6204-4b52-a74e-ae5dfb00e9b3/test-curl-6908-140273651942096/0.log" Jul 02 20:58:37.649480 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.649452 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6908-140273651942096"] Jul 02 20:58:37.651657 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.651632 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6908-140273651942096"] Jul 02 20:58:37.715090 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.715062 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lg7n\" (UniqueName: \"kubernetes.io/projected/2a3717a1-6204-4b52-a74e-ae5dfb00e9b3-kube-api-access-9lg7n\") pod \"2a3717a1-6204-4b52-a74e-ae5dfb00e9b3\" (UID: \"2a3717a1-6204-4b52-a74e-ae5dfb00e9b3\") " Jul 02 20:58:37.717295 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.717263 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3717a1-6204-4b52-a74e-ae5dfb00e9b3-kube-api-access-9lg7n" (OuterVolumeSpecName: "kube-api-access-9lg7n") pod "2a3717a1-6204-4b52-a74e-ae5dfb00e9b3" (UID: "2a3717a1-6204-4b52-a74e-ae5dfb00e9b3"). InnerVolumeSpecName "kube-api-access-9lg7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:37.774817 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.774750 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6908-140273651942208"] Jul 02 20:58:37.775154 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.775141 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a3717a1-6204-4b52-a74e-ae5dfb00e9b3" containerName="test-curl-6908-140273651942096" Jul 02 20:58:37.775192 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.775157 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a3717a1-6204-4b52-a74e-ae5dfb00e9b3" containerName="test-curl-6908-140273651942096" Jul 02 20:58:37.775274 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.775254 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a3717a1-6204-4b52-a74e-ae5dfb00e9b3" containerName="test-curl-6908-140273651942096" Jul 02 20:58:37.778350 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.778335 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651942208" Jul 02 20:58:37.783764 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.783742 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6908-140273651942208"] Jul 02 20:58:37.815763 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.815742 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lg7n\" (UniqueName: \"kubernetes.io/projected/2a3717a1-6204-4b52-a74e-ae5dfb00e9b3-kube-api-access-9lg7n\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:37.916312 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:37.916275 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2nsr\" (UniqueName: \"kubernetes.io/projected/c44adef3-07ff-4c54-a292-7cdc0e713d7d-kube-api-access-c2nsr\") pod \"test-curl-6908-140273651942208\" (UID: \"c44adef3-07ff-4c54-a292-7cdc0e713d7d\") " pod="opendatahub/test-curl-6908-140273651942208" Jul 02 20:58:38.017689 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:38.017647 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2nsr\" (UniqueName: \"kubernetes.io/projected/c44adef3-07ff-4c54-a292-7cdc0e713d7d-kube-api-access-c2nsr\") pod \"test-curl-6908-140273651942208\" (UID: \"c44adef3-07ff-4c54-a292-7cdc0e713d7d\") " pod="opendatahub/test-curl-6908-140273651942208" Jul 02 20:58:38.025772 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:38.025716 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2nsr\" (UniqueName: \"kubernetes.io/projected/c44adef3-07ff-4c54-a292-7cdc0e713d7d-kube-api-access-c2nsr\") pod \"test-curl-6908-140273651942208\" (UID: \"c44adef3-07ff-4c54-a292-7cdc0e713d7d\") " pod="opendatahub/test-curl-6908-140273651942208" Jul 02 20:58:38.088234 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:38.088182 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651942208" Jul 02 20:58:38.217370 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:38.217342 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6908-140273651942208"] Jul 02 20:58:38.218934 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:58:38.218904 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc44adef3_07ff_4c54_a292_7cdc0e713d7d.slice/crio-b7c2b072dac8f109dbf81b4026957ed274db0f979280441b80ee9e379d56d071 WatchSource:0}: Error finding container b7c2b072dac8f109dbf81b4026957ed274db0f979280441b80ee9e379d56d071: Status 404 returned error can't find the container with id b7c2b072dac8f109dbf81b4026957ed274db0f979280441b80ee9e379d56d071 Jul 02 20:58:38.481471 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:38.481448 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651942096" Jul 02 20:58:38.481676 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:38.481452 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd641f9186c2e4a38bae75f496fbcb7c5c2f76bde56ac0a7c66d201b86c36af6" Jul 02 20:58:38.482627 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:38.482603 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6908-140273651942208" event={"ID":"c44adef3-07ff-4c54-a292-7cdc0e713d7d","Type":"ContainerStarted","Data":"b7c2b072dac8f109dbf81b4026957ed274db0f979280441b80ee9e379d56d071"} Jul 02 20:58:38.491877 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:38.491852 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273651942096\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="2a3717a1-6204-4b52-a74e-ae5dfb00e9b3" pod="opendatahub/test-curl-6908-140273651942096" Jul 02 20:58:39.495857 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:39.495822 2565 generic.go:356] "Generic (PLEG): container finished" podID="c44adef3-07ff-4c54-a292-7cdc0e713d7d" containerID="2911819e3a57352db5b1d9680b260ad7509fc1e6ff928655cf2b5753a84ffb39" exitCode=0 Jul 02 20:58:39.496258 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:39.495907 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6908-140273651942208" event={"ID":"c44adef3-07ff-4c54-a292-7cdc0e713d7d","Type":"ContainerDied","Data":"2911819e3a57352db5b1d9680b260ad7509fc1e6ff928655cf2b5753a84ffb39"} Jul 02 20:58:39.498148 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:39.498122 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273651942096\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="2a3717a1-6204-4b52-a74e-ae5dfb00e9b3" pod="opendatahub/test-curl-6908-140273651942096" Jul 02 20:58:39.571398 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:39.571365 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a3717a1-6204-4b52-a74e-ae5dfb00e9b3" path="/var/lib/kubelet/pods/2a3717a1-6204-4b52-a74e-ae5dfb00e9b3/volumes" Jul 02 20:58:40.631259 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.631232 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651942208" Jul 02 20:58:40.651148 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.651126 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6908-140273651942208_c44adef3-07ff-4c54-a292-7cdc0e713d7d/test-curl-6908-140273651942208/0.log" Jul 02 20:58:40.675238 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.675198 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6908-140273651942208"] Jul 02 20:58:40.679015 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.678992 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6908-140273651942208"] Jul 02 20:58:40.742758 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.742733 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2nsr\" (UniqueName: \"kubernetes.io/projected/c44adef3-07ff-4c54-a292-7cdc0e713d7d-kube-api-access-c2nsr\") pod \"c44adef3-07ff-4c54-a292-7cdc0e713d7d\" (UID: \"c44adef3-07ff-4c54-a292-7cdc0e713d7d\") " Jul 02 20:58:40.744869 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.744842 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44adef3-07ff-4c54-a292-7cdc0e713d7d-kube-api-access-c2nsr" (OuterVolumeSpecName: "kube-api-access-c2nsr") pod "c44adef3-07ff-4c54-a292-7cdc0e713d7d" (UID: "c44adef3-07ff-4c54-a292-7cdc0e713d7d"). InnerVolumeSpecName "kube-api-access-c2nsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:40.802038 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.801974 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6908-140273651941872"] Jul 02 20:58:40.802376 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.802363 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44adef3-07ff-4c54-a292-7cdc0e713d7d" containerName="test-curl-6908-140273651942208" Jul 02 20:58:40.802422 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.802378 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44adef3-07ff-4c54-a292-7cdc0e713d7d" containerName="test-curl-6908-140273651942208" Jul 02 20:58:40.802461 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.802446 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c44adef3-07ff-4c54-a292-7cdc0e713d7d" containerName="test-curl-6908-140273651942208" Jul 02 20:58:40.805308 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.805293 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651941872" Jul 02 20:58:40.813382 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.813357 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6908-140273651941872"] Jul 02 20:58:40.843916 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.843891 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2nsr\" (UniqueName: \"kubernetes.io/projected/c44adef3-07ff-4c54-a292-7cdc0e713d7d-kube-api-access-c2nsr\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:40.945020 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:40.944982 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82w2j\" (UniqueName: \"kubernetes.io/projected/7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8-kube-api-access-82w2j\") pod \"test-curl-6908-140273651941872\" (UID: \"7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8\") " pod="opendatahub/test-curl-6908-140273651941872" Jul 02 20:58:41.046564 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:41.046531 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82w2j\" (UniqueName: \"kubernetes.io/projected/7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8-kube-api-access-82w2j\") pod \"test-curl-6908-140273651941872\" (UID: \"7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8\") " pod="opendatahub/test-curl-6908-140273651941872" Jul 02 20:58:41.055396 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:41.055327 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82w2j\" (UniqueName: \"kubernetes.io/projected/7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8-kube-api-access-82w2j\") pod \"test-curl-6908-140273651941872\" (UID: \"7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8\") " pod="opendatahub/test-curl-6908-140273651941872" Jul 02 20:58:41.114954 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:41.114927 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651941872" Jul 02 20:58:41.251732 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:41.251681 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6908-140273651941872"] Jul 02 20:58:41.253572 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:58:41.253546 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ad6e485_6bbb_4f3d_9c04_be613ce0c8c8.slice/crio-dcb567dd838ab6b0bbcee125d63d97c4f3d1324b4f2c95f9378c8fd62c159acf WatchSource:0}: Error finding container dcb567dd838ab6b0bbcee125d63d97c4f3d1324b4f2c95f9378c8fd62c159acf: Status 404 returned error can't find the container with id dcb567dd838ab6b0bbcee125d63d97c4f3d1324b4f2c95f9378c8fd62c159acf Jul 02 20:58:41.506907 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:41.506874 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6908-140273651941872" event={"ID":"7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8","Type":"ContainerStarted","Data":"dcb567dd838ab6b0bbcee125d63d97c4f3d1324b4f2c95f9378c8fd62c159acf"} Jul 02 20:58:41.508110 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:41.508089 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7c2b072dac8f109dbf81b4026957ed274db0f979280441b80ee9e379d56d071" Jul 02 20:58:41.508256 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:41.508124 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651942208" Jul 02 20:58:41.520571 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:41.520547 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273651942208\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="c44adef3-07ff-4c54-a292-7cdc0e713d7d" pod="opendatahub/test-curl-6908-140273651942208" Jul 02 20:58:41.570134 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:41.570106 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273651942208\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="c44adef3-07ff-4c54-a292-7cdc0e713d7d" pod="opendatahub/test-curl-6908-140273651942208" Jul 02 20:58:41.572041 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:41.572017 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c44adef3-07ff-4c54-a292-7cdc0e713d7d" path="/var/lib/kubelet/pods/c44adef3-07ff-4c54-a292-7cdc0e713d7d/volumes" Jul 02 20:58:42.514911 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:42.514879 2565 generic.go:356] "Generic (PLEG): container finished" podID="7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8" containerID="9f046ca4d9982032da3dadb6f6b4eb6d88cc9aa93d3e495b902a44ec7ce5be21" exitCode=0 Jul 02 20:58:42.515340 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:42.514959 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6908-140273651941872" event={"ID":"7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8","Type":"ContainerDied","Data":"9f046ca4d9982032da3dadb6f6b4eb6d88cc9aa93d3e495b902a44ec7ce5be21"} Jul 02 20:58:43.657664 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.657638 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651941872" Jul 02 20:58:43.679117 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.679095 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6908-140273651941872_7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8/test-curl-6908-140273651941872/0.log" Jul 02 20:58:43.708486 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.708451 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6908-140273651941872"] Jul 02 20:58:43.710084 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.710058 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6908-140273651941872"] Jul 02 20:58:43.772592 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.772528 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82w2j\" (UniqueName: \"kubernetes.io/projected/7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8-kube-api-access-82w2j\") pod \"7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8\" (UID: \"7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8\") " Jul 02 20:58:43.774682 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.774661 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8-kube-api-access-82w2j" (OuterVolumeSpecName: "kube-api-access-82w2j") pod "7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8" (UID: "7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8"). InnerVolumeSpecName "kube-api-access-82w2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:43.868627 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.868598 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664"] Jul 02 20:58:43.868994 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.868982 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8" containerName="test-curl-6908-140273651941872" Jul 02 20:58:43.869044 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.868996 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8" containerName="test-curl-6908-140273651941872" Jul 02 20:58:43.869082 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.869070 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8" containerName="test-curl-6908-140273651941872" Jul 02 20:58:43.872191 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.872176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664" Jul 02 20:58:43.873453 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.873431 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82w2j\" (UniqueName: \"kubernetes.io/projected/7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8-kube-api-access-82w2j\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:43.883617 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.883597 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664"] Jul 02 20:58:43.974776 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:43.974735 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wf5x\" (UniqueName: \"kubernetes.io/projected/7c32e056-1c8b-40d7-811e-49c1c94eeb34-kube-api-access-8wf5x\") pod \"test-curl-6908-140273652107664\" (UID: \"7c32e056-1c8b-40d7-811e-49c1c94eeb34\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664" Jul 02 20:58:44.076109 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:44.076016 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wf5x\" (UniqueName: \"kubernetes.io/projected/7c32e056-1c8b-40d7-811e-49c1c94eeb34-kube-api-access-8wf5x\") pod \"test-curl-6908-140273652107664\" (UID: \"7c32e056-1c8b-40d7-811e-49c1c94eeb34\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664" Jul 02 20:58:44.084501 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:44.084470 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wf5x\" (UniqueName: \"kubernetes.io/projected/7c32e056-1c8b-40d7-811e-49c1c94eeb34-kube-api-access-8wf5x\") pod \"test-curl-6908-140273652107664\" (UID: \"7c32e056-1c8b-40d7-811e-49c1c94eeb34\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664" Jul 02 20:58:44.181558 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:44.181523 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664" Jul 02 20:58:44.310869 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:44.310845 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664"] Jul 02 20:58:44.312728 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:58:44.312705 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c32e056_1c8b_40d7_811e_49c1c94eeb34.slice/crio-47e22778584217c902ad6b4d62754e18e59401fe4d593ded41fc340cf3c2b12a WatchSource:0}: Error finding container 47e22778584217c902ad6b4d62754e18e59401fe4d593ded41fc340cf3c2b12a: Status 404 returned error can't find the container with id 47e22778584217c902ad6b4d62754e18e59401fe4d593ded41fc340cf3c2b12a Jul 02 20:58:44.526648 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:44.526618 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcb567dd838ab6b0bbcee125d63d97c4f3d1324b4f2c95f9378c8fd62c159acf" Jul 02 20:58:44.526648 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:44.526638 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6908-140273651941872" Jul 02 20:58:44.527855 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:44.527827 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664" event={"ID":"7c32e056-1c8b-40d7-811e-49c1c94eeb34","Type":"ContainerStarted","Data":"47e22778584217c902ad6b4d62754e18e59401fe4d593ded41fc340cf3c2b12a"} Jul 02 20:58:44.537812 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:44.537787 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273651941872\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8" pod="opendatahub/test-curl-6908-140273651941872" Jul 02 20:58:44.828936 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:44.828895 2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" podUID="48b206b4-fbe9-4102-96c3-52cd2fc8eb01" containerName="main" containerID="cri-o://2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51" gracePeriod=45 Jul 02 20:58:45.079900 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.079872 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:58:45.100094 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:45.100054 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273651941872\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8" pod="opendatahub/test-curl-6908-140273651941872" Jul 02 20:58:45.184696 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.184664 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-model-cache\") pod \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " Jul 02 20:58:45.184846 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.184736 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tmp-dir\") pod \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " Jul 02 20:58:45.184846 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.184794 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-home\") pod \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " Jul 02 20:58:45.184945 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.184843 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzwnd\" (UniqueName: \"kubernetes.io/projected/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-kube-api-access-zzwnd\") pod \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " Jul 02 20:58:45.184945 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.184869 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-dshm\") pod \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " Jul 02 20:58:45.184945 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.184906 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tls-certs\") pod \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\" (UID: \"48b206b4-fbe9-4102-96c3-52cd2fc8eb01\") " Jul 02 20:58:45.185098 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.184951 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-model-cache" (OuterVolumeSpecName: "model-cache") pod "48b206b4-fbe9-4102-96c3-52cd2fc8eb01" (UID: "48b206b4-fbe9-4102-96c3-52cd2fc8eb01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:58:45.185098 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.185027 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48b206b4-fbe9-4102-96c3-52cd2fc8eb01" (UID: "48b206b4-fbe9-4102-96c3-52cd2fc8eb01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:58:45.185098 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.185038 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-home" (OuterVolumeSpecName: "home") pod "48b206b4-fbe9-4102-96c3-52cd2fc8eb01" (UID: "48b206b4-fbe9-4102-96c3-52cd2fc8eb01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:58:45.185282 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.185191 2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-home\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:45.185282 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.185236 2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-model-cache\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:45.185282 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.185247 2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tmp-dir\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:45.187096 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.187066 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-dshm" (OuterVolumeSpecName: "dshm") pod "48b206b4-fbe9-4102-96c3-52cd2fc8eb01" (UID: "48b206b4-fbe9-4102-96c3-52cd2fc8eb01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 20:58:45.187259 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.187094 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48b206b4-fbe9-4102-96c3-52cd2fc8eb01" (UID: "48b206b4-fbe9-4102-96c3-52cd2fc8eb01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 20:58:45.187259 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.187141 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-kube-api-access-zzwnd" (OuterVolumeSpecName: "kube-api-access-zzwnd") pod "48b206b4-fbe9-4102-96c3-52cd2fc8eb01" (UID: "48b206b4-fbe9-4102-96c3-52cd2fc8eb01"). InnerVolumeSpecName "kube-api-access-zzwnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:45.286478 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.286397 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzwnd\" (UniqueName: \"kubernetes.io/projected/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-kube-api-access-zzwnd\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:45.286478 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.286425 2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-dshm\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:45.286478 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.286435 2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48b206b4-fbe9-4102-96c3-52cd2fc8eb01-tls-certs\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:45.535661 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.535627 2565 generic.go:356] "Generic (PLEG): container finished" podID="7c32e056-1c8b-40d7-811e-49c1c94eeb34" containerID="d2530e831c80849949883da7cfd95605753d46c4b69767429f1256581eaa73bf" exitCode=6 Jul 02 20:58:45.535822 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.535719 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664" event={"ID":"7c32e056-1c8b-40d7-811e-49c1c94eeb34","Type":"ContainerDied","Data":"d2530e831c80849949883da7cfd95605753d46c4b69767429f1256581eaa73bf"} Jul 02 20:58:45.536943 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.536876 2565 generic.go:356] "Generic (PLEG): container finished" podID="48b206b4-fbe9-4102-96c3-52cd2fc8eb01" containerID="2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51" exitCode=0 Jul 02 20:58:45.536943 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.536923 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" Jul 02 20:58:45.537117 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.536926 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" event={"ID":"48b206b4-fbe9-4102-96c3-52cd2fc8eb01","Type":"ContainerDied","Data":"2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51"} Jul 02 20:58:45.537117 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.537042 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt" event={"ID":"48b206b4-fbe9-4102-96c3-52cd2fc8eb01","Type":"ContainerDied","Data":"c77fc57329f59d4abbceacffa8b50de4fdbb1be6c5e5acdfcfec8a3fd2916632"} Jul 02 20:58:45.537117 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.537069 2565 scope.go:117] "RemoveContainer" containerID="2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51" Jul 02 20:58:45.546657 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.546641 2565 scope.go:117] "RemoveContainer" containerID="2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51" Jul 02 20:58:45.546919 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:45.546900 2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51\": container with ID starting with 2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51 not found: ID does not exist" containerID="2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51" Jul 02 20:58:45.546999 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.546926 2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51"} err="failed to get container status \"2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51\": rpc error: code = NotFound desc = could not find container \"2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51\": container with ID starting with 2e8182d1cd1f1ebd13025183b68aa218093187e1828464dc429c0ee4cb9d8e51 not found: ID does not exist" Jul 02 20:58:45.552406 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:45.552384 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273651941872\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8" pod="opendatahub/test-curl-6908-140273651941872" Jul 02 20:58:45.564246 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.564225 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt"] Jul 02 20:58:45.567796 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:45.567772 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273651941872\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8" pod="opendatahub/test-curl-6908-140273651941872" Jul 02 20:58:45.572899 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.572873 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8" path="/var/lib/kubelet/pods/7ad6e485-6bbb-4f3d-9c04-be613ce0c8c8/volumes" Jul 02 20:58:45.573177 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:45.573160 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/rate-test-model-bf7336-kserve-895458579-5bdqt"] Jul 02 20:58:46.679089 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.679062 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664" Jul 02 20:58:46.698632 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.698609 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-shared-a-bf7336_test-curl-6908-140273652107664_7c32e056-1c8b-40d7-811e-49c1c94eeb34/test-curl-6908-140273652107664/0.log" Jul 02 20:58:46.724019 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.723988 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664"] Jul 02 20:58:46.727499 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.727476 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664"] Jul 02 20:58:46.802045 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.801961 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8wf5x\" (UniqueName: \"kubernetes.io/projected/7c32e056-1c8b-40d7-811e-49c1c94eeb34-kube-api-access-8wf5x\") pod \"7c32e056-1c8b-40d7-811e-49c1c94eeb34\" (UID: \"7c32e056-1c8b-40d7-811e-49c1c94eeb34\") " Jul 02 20:58:46.804304 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.804278 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c32e056-1c8b-40d7-811e-49c1c94eeb34-kube-api-access-8wf5x" (OuterVolumeSpecName: "kube-api-access-8wf5x") pod "7c32e056-1c8b-40d7-811e-49c1c94eeb34" (UID: "7c32e056-1c8b-40d7-811e-49c1c94eeb34"). InnerVolumeSpecName "kube-api-access-8wf5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:46.856847 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.856814 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104"] Jul 02 20:58:46.857230 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.857194 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48b206b4-fbe9-4102-96c3-52cd2fc8eb01" containerName="main" Jul 02 20:58:46.857230 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.857221 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="48b206b4-fbe9-4102-96c3-52cd2fc8eb01" containerName="main" Jul 02 20:58:46.857304 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.857234 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c32e056-1c8b-40d7-811e-49c1c94eeb34" containerName="test-curl-6908-140273652107664" Jul 02 20:58:46.857304 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.857242 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c32e056-1c8b-40d7-811e-49c1c94eeb34" containerName="test-curl-6908-140273652107664" Jul 02 20:58:46.857304 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.857298 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c32e056-1c8b-40d7-811e-49c1c94eeb34" containerName="test-curl-6908-140273652107664" Jul 02 20:58:46.857402 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.857307 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="48b206b4-fbe9-4102-96c3-52cd2fc8eb01" containerName="main" Jul 02 20:58:46.860063 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.860049 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104" Jul 02 20:58:46.865517 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.865494 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104"] Jul 02 20:58:46.903162 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:46.903135 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wf5x\" (UniqueName: \"kubernetes.io/projected/7c32e056-1c8b-40d7-811e-49c1c94eeb34-kube-api-access-8wf5x\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:47.004039 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:47.004010 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnfj\" (UniqueName: \"kubernetes.io/projected/0d3cab49-7991-420b-bcca-4dd47eaf01c8-kube-api-access-jrnfj\") pod \"test-curl-6908-140273652109104\" (UID: \"0d3cab49-7991-420b-bcca-4dd47eaf01c8\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104" Jul 02 20:58:47.104941 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:47.104855 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnfj\" (UniqueName: \"kubernetes.io/projected/0d3cab49-7991-420b-bcca-4dd47eaf01c8-kube-api-access-jrnfj\") pod \"test-curl-6908-140273652109104\" (UID: \"0d3cab49-7991-420b-bcca-4dd47eaf01c8\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104" Jul 02 20:58:47.113491 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:47.113471 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnfj\" (UniqueName: \"kubernetes.io/projected/0d3cab49-7991-420b-bcca-4dd47eaf01c8-kube-api-access-jrnfj\") pod \"test-curl-6908-140273652109104\" (UID: \"0d3cab49-7991-420b-bcca-4dd47eaf01c8\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104" Jul 02 20:58:47.169997 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:47.169963 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104" Jul 02 20:58:47.301634 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:47.301607 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104"] Jul 02 20:58:47.303592 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:58:47.303569 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d3cab49_7991_420b_bcca_4dd47eaf01c8.slice/crio-dfe8c3600055abd861e35611f38ec3d10db965599d80b8b4103a22a06f571386 WatchSource:0}: Error finding container dfe8c3600055abd861e35611f38ec3d10db965599d80b8b4103a22a06f571386: Status 404 returned error can't find the container with id dfe8c3600055abd861e35611f38ec3d10db965599d80b8b4103a22a06f571386 Jul 02 20:58:47.549091 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:47.549057 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664" Jul 02 20:58:47.549313 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:47.549067 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47e22778584217c902ad6b4d62754e18e59401fe4d593ded41fc340cf3c2b12a" Jul 02 20:58:47.550331 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:47.550295 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104" event={"ID":"0d3cab49-7991-420b-bcca-4dd47eaf01c8","Type":"ContainerStarted","Data":"dfe8c3600055abd861e35611f38ec3d10db965599d80b8b4103a22a06f571386"} Jul 02 20:58:47.559444 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:47.559414 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273652107664\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-bf7336\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="7c32e056-1c8b-40d7-811e-49c1c94eeb34" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652107664" Jul 02 20:58:47.571478 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:47.571454 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48b206b4-fbe9-4102-96c3-52cd2fc8eb01" path="/var/lib/kubelet/pods/48b206b4-fbe9-4102-96c3-52cd2fc8eb01/volumes" Jul 02 20:58:47.571808 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:47.571793 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c32e056-1c8b-40d7-811e-49c1c94eeb34" path="/var/lib/kubelet/pods/7c32e056-1c8b-40d7-811e-49c1c94eeb34/volumes" Jul 02 20:58:48.557433 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:48.557329 2565 generic.go:356] "Generic (PLEG): container finished" podID="0d3cab49-7991-420b-bcca-4dd47eaf01c8" containerID="d132e3c6cc772e8ba3c6848a77ab4e4b2a0675c64cd0ecb4306a4c252ac856f3" exitCode=6 Jul 02 20:58:48.557433 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:48.557390 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104" event={"ID":"0d3cab49-7991-420b-bcca-4dd47eaf01c8","Type":"ContainerDied","Data":"d132e3c6cc772e8ba3c6848a77ab4e4b2a0675c64cd0ecb4306a4c252ac856f3"} Jul 02 20:58:49.706305 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.706270 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104" Jul 02 20:58:49.728515 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.728496 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-shared-a-bf7336_test-curl-6908-140273652109104_0d3cab49-7991-420b-bcca-4dd47eaf01c8/test-curl-6908-140273652109104/0.log" Jul 02 20:58:49.751994 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.751963 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104"] Jul 02 20:58:49.755670 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.755650 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104"] Jul 02 20:58:49.833457 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.833390 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrnfj\" (UniqueName: \"kubernetes.io/projected/0d3cab49-7991-420b-bcca-4dd47eaf01c8-kube-api-access-jrnfj\") pod \"0d3cab49-7991-420b-bcca-4dd47eaf01c8\" (UID: \"0d3cab49-7991-420b-bcca-4dd47eaf01c8\") " Jul 02 20:58:49.835543 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.835520 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3cab49-7991-420b-bcca-4dd47eaf01c8-kube-api-access-jrnfj" (OuterVolumeSpecName: "kube-api-access-jrnfj") pod "0d3cab49-7991-420b-bcca-4dd47eaf01c8" (UID: "0d3cab49-7991-420b-bcca-4dd47eaf01c8"). InnerVolumeSpecName "kube-api-access-jrnfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:49.883383 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.883356 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464"] Jul 02 20:58:49.883710 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.883698 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3cab49-7991-420b-bcca-4dd47eaf01c8" containerName="test-curl-6908-140273652109104" Jul 02 20:58:49.883762 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.883712 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3cab49-7991-420b-bcca-4dd47eaf01c8" containerName="test-curl-6908-140273652109104" Jul 02 20:58:49.883800 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.883790 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3cab49-7991-420b-bcca-4dd47eaf01c8" containerName="test-curl-6908-140273652109104" Jul 02 20:58:49.886548 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.886526 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464" Jul 02 20:58:49.893014 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.892991 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464"] Jul 02 20:58:49.934894 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:49.934864 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrnfj\" (UniqueName: \"kubernetes.io/projected/0d3cab49-7991-420b-bcca-4dd47eaf01c8-kube-api-access-jrnfj\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:50.035511 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:50.035473 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j24b\" (UniqueName: \"kubernetes.io/projected/c315fa35-b59e-4985-bf01-0867b548a229-kube-api-access-5j24b\") pod \"test-curl-6908-140273652108464\" (UID: \"c315fa35-b59e-4985-bf01-0867b548a229\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464" Jul 02 20:58:50.136320 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:50.136198 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j24b\" (UniqueName: \"kubernetes.io/projected/c315fa35-b59e-4985-bf01-0867b548a229-kube-api-access-5j24b\") pod \"test-curl-6908-140273652108464\" (UID: \"c315fa35-b59e-4985-bf01-0867b548a229\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464" Jul 02 20:58:50.150393 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:50.150359 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j24b\" (UniqueName: \"kubernetes.io/projected/c315fa35-b59e-4985-bf01-0867b548a229-kube-api-access-5j24b\") pod \"test-curl-6908-140273652108464\" (UID: \"c315fa35-b59e-4985-bf01-0867b548a229\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464" Jul 02 20:58:50.196204 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:50.196176 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464" Jul 02 20:58:50.358133 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:50.358112 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464"] Jul 02 20:58:50.360300 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:58:50.360270 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc315fa35_b59e_4985_bf01_0867b548a229.slice/crio-f74845d3db174516500ae78a2212a03a554fcdc239ef2a75877a73b96432f275 WatchSource:0}: Error finding container f74845d3db174516500ae78a2212a03a554fcdc239ef2a75877a73b96432f275: Status 404 returned error can't find the container with id f74845d3db174516500ae78a2212a03a554fcdc239ef2a75877a73b96432f275 Jul 02 20:58:50.568173 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:50.568138 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104" Jul 02 20:58:50.568373 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:50.568176 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfe8c3600055abd861e35611f38ec3d10db965599d80b8b4103a22a06f571386" Jul 02 20:58:50.569387 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:50.569361 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464" event={"ID":"c315fa35-b59e-4985-bf01-0867b548a229","Type":"ContainerStarted","Data":"f74845d3db174516500ae78a2212a03a554fcdc239ef2a75877a73b96432f275"} Jul 02 20:58:50.578489 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:50.578464 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273652109104\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-bf7336\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="0d3cab49-7991-420b-bcca-4dd47eaf01c8" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104" Jul 02 20:58:51.570609 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:51.570570 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273652109104\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-bf7336\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="0d3cab49-7991-420b-bcca-4dd47eaf01c8" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652109104" Jul 02 20:58:51.571826 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:51.571802 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3cab49-7991-420b-bcca-4dd47eaf01c8" path="/var/lib/kubelet/pods/0d3cab49-7991-420b-bcca-4dd47eaf01c8/volumes" Jul 02 20:58:51.574755 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:51.574728 2565 generic.go:356] "Generic (PLEG): container finished" podID="c315fa35-b59e-4985-bf01-0867b548a229" containerID="45d1bdbce9aad7f5bd40d1f31a34cf40c2b8ec59fdca7e4e13458253510bf3f8" exitCode=6 Jul 02 20:58:51.574887 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:51.574789 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464" event={"ID":"c315fa35-b59e-4985-bf01-0867b548a229","Type":"ContainerDied","Data":"45d1bdbce9aad7f5bd40d1f31a34cf40c2b8ec59fdca7e4e13458253510bf3f8"} Jul 02 20:58:52.718717 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.718694 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464" Jul 02 20:58:52.738417 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.738381 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-shared-a-bf7336_test-curl-6908-140273652108464_c315fa35-b59e-4985-bf01-0867b548a229/test-curl-6908-140273652108464/0.log" Jul 02 20:58:52.762935 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.762905 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464"] Jul 02 20:58:52.768502 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.768481 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464"] Jul 02 20:58:52.862047 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.861977 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j24b\" (UniqueName: \"kubernetes.io/projected/c315fa35-b59e-4985-bf01-0867b548a229-kube-api-access-5j24b\") pod \"c315fa35-b59e-4985-bf01-0867b548a229\" (UID: \"c315fa35-b59e-4985-bf01-0867b548a229\") " Jul 02 20:58:52.863907 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.863879 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c315fa35-b59e-4985-bf01-0867b548a229-kube-api-access-5j24b" (OuterVolumeSpecName: "kube-api-access-5j24b") pod "c315fa35-b59e-4985-bf01-0867b548a229" (UID: "c315fa35-b59e-4985-bf01-0867b548a229"). InnerVolumeSpecName "kube-api-access-5j24b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:52.910872 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.910839 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368"] Jul 02 20:58:52.911229 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.911204 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c315fa35-b59e-4985-bf01-0867b548a229" containerName="test-curl-6908-140273652108464" Jul 02 20:58:52.911276 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.911231 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c315fa35-b59e-4985-bf01-0867b548a229" containerName="test-curl-6908-140273652108464" Jul 02 20:58:52.911311 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.911291 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c315fa35-b59e-4985-bf01-0867b548a229" containerName="test-curl-6908-140273652108464" Jul 02 20:58:52.916288 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.914805 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368" Jul 02 20:58:52.921683 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.921655 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368"] Jul 02 20:58:52.963182 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:52.963156 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j24b\" (UniqueName: \"kubernetes.io/projected/c315fa35-b59e-4985-bf01-0867b548a229-kube-api-access-5j24b\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:53.064278 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:53.064245 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47mf\" (UniqueName: \"kubernetes.io/projected/781b5a50-4bb8-4bb5-8b21-0727acdf306b-kube-api-access-p47mf\") pod \"test-curl-6908-140273651856368\" (UID: \"781b5a50-4bb8-4bb5-8b21-0727acdf306b\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368" Jul 02 20:58:53.165066 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:53.165036 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p47mf\" (UniqueName: \"kubernetes.io/projected/781b5a50-4bb8-4bb5-8b21-0727acdf306b-kube-api-access-p47mf\") pod \"test-curl-6908-140273651856368\" (UID: \"781b5a50-4bb8-4bb5-8b21-0727acdf306b\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368" Jul 02 20:58:53.174323 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:53.174294 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47mf\" (UniqueName: \"kubernetes.io/projected/781b5a50-4bb8-4bb5-8b21-0727acdf306b-kube-api-access-p47mf\") pod \"test-curl-6908-140273651856368\" (UID: \"781b5a50-4bb8-4bb5-8b21-0727acdf306b\") " pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368" Jul 02 20:58:53.228876 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:53.228851 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368" Jul 02 20:58:53.359340 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:53.359318 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368"] Jul 02 20:58:53.361502 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:58:53.361477 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod781b5a50_4bb8_4bb5_8b21_0727acdf306b.slice/crio-ef4b9b929b215b9beefbebe0039338be46df0d8ad3ad80da932554ae4540a95b WatchSource:0}: Error finding container ef4b9b929b215b9beefbebe0039338be46df0d8ad3ad80da932554ae4540a95b: Status 404 returned error can't find the container with id ef4b9b929b215b9beefbebe0039338be46df0d8ad3ad80da932554ae4540a95b Jul 02 20:58:53.571024 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:53.570935 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c315fa35-b59e-4985-bf01-0867b548a229" path="/var/lib/kubelet/pods/c315fa35-b59e-4985-bf01-0867b548a229/volumes" Jul 02 20:58:53.585619 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:53.585592 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273652108464" Jul 02 20:58:53.585736 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:53.585624 2565 scope.go:117] "RemoveContainer" containerID="45d1bdbce9aad7f5bd40d1f31a34cf40c2b8ec59fdca7e4e13458253510bf3f8" Jul 02 20:58:53.587047 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:53.587019 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368" event={"ID":"781b5a50-4bb8-4bb5-8b21-0727acdf306b","Type":"ContainerStarted","Data":"ef4b9b929b215b9beefbebe0039338be46df0d8ad3ad80da932554ae4540a95b"} Jul 02 20:58:54.594023 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:54.593986 2565 generic.go:356] "Generic (PLEG): container finished" podID="781b5a50-4bb8-4bb5-8b21-0727acdf306b" containerID="7489bcc9676bcd8bc414680dee6b6dbfac4b52ab3145f7c2445e07675735711a" exitCode=6 Jul 02 20:58:54.594412 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:54.594045 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368" event={"ID":"781b5a50-4bb8-4bb5-8b21-0727acdf306b","Type":"ContainerDied","Data":"7489bcc9676bcd8bc414680dee6b6dbfac4b52ab3145f7c2445e07675735711a"} Jul 02 20:58:55.737149 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:55.737126 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368" Jul 02 20:58:55.757376 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:55.757347 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/ai-tenant-e2e-shared-a-bf7336_test-curl-6908-140273651856368_781b5a50-4bb8-4bb5-8b21-0727acdf306b/test-curl-6908-140273651856368/0.log" Jul 02 20:58:55.782592 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:55.782556 2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368"] Jul 02 20:58:55.785478 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:55.785454 2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368"] Jul 02 20:58:55.887773 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:55.887684 2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p47mf\" (UniqueName: \"kubernetes.io/projected/781b5a50-4bb8-4bb5-8b21-0727acdf306b-kube-api-access-p47mf\") pod \"781b5a50-4bb8-4bb5-8b21-0727acdf306b\" (UID: \"781b5a50-4bb8-4bb5-8b21-0727acdf306b\") " Jul 02 20:58:55.889815 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:55.889781 2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781b5a50-4bb8-4bb5-8b21-0727acdf306b-kube-api-access-p47mf" (OuterVolumeSpecName: "kube-api-access-p47mf") pod "781b5a50-4bb8-4bb5-8b21-0727acdf306b" (UID: "781b5a50-4bb8-4bb5-8b21-0727acdf306b"). InnerVolumeSpecName "kube-api-access-p47mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 20:58:55.988620 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:55.988586 2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p47mf\" (UniqueName: \"kubernetes.io/projected/781b5a50-4bb8-4bb5-8b21-0727acdf306b-kube-api-access-p47mf\") on node \"ip-10-0-132-15.ec2.internal\" DevicePath \"\"" Jul 02 20:58:56.002480 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:56.002449 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:56.406966 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:56.406909 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:56.506603 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:56.506575 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:56.605852 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:56.605826 2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef4b9b929b215b9beefbebe0039338be46df0d8ad3ad80da932554ae4540a95b" Jul 02 20:58:56.605988 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:56.605894 2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368" Jul 02 20:58:56.616328 ip-10-0-132-15 kubenswrapper[2565]: E0702 20:58:56.616289 2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6908-140273651856368\" is forbidden: User \"system:node:ip-10-0-132-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"ai-tenant-e2e-shared-a-bf7336\": no relationship found between node 'ip-10-0-132-15.ec2.internal' and this object" podUID="781b5a50-4bb8-4bb5-8b21-0727acdf306b" pod="ai-tenant-e2e-shared-a-bf7336/test-curl-6908-140273651856368" Jul 02 20:58:56.649666 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:56.649638 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:58:57.571375 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:58:57.571344 2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="781b5a50-4bb8-4bb5-8b21-0727acdf306b" path="/var/lib/kubelet/pods/781b5a50-4bb8-4bb5-8b21-0727acdf306b/volumes" Jul 02 20:59:02.234085 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:02.234049 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:02.300854 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:02.300829 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:02.571913 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:02.571830 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:02.772310 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:02.772277 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:02.918493 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:02.918470 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:03.314077 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:03.313985 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:03.415373 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:03.415319 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:03.517710 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:03.517664 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:03.635512 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:03.635413 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:09.155631 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:09.155587 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:09.257423 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:09.257370 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:09.412998 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:09.412917 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:09.650254 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:09.650227 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:14.905399 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:14.905368 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-55f547546d-456dr_5a7c2b51-b00f-41e8-92d3-d97e2e13d53e/maas-api/0.log" Jul 02 20:59:15.606750 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:15.606722 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b478dcdf-4p54h_3929840f-933b-4207-8684-5a10e32135c0/copy-manifests/0.log" Jul 02 20:59:15.614829 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:15.614811 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b478dcdf-4p54h_3929840f-933b-4207-8684-5a10e32135c0/manager/0.log" Jul 02 20:59:15.991843 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:15.991818 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-h9t5q_98c73987-16d5-4082-bff8-ebfbf34c0406/postgres/0.log" Jul 02 20:59:17.598367 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:17.598338 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-d9hxp_d6b2d93d-1f86-4f70-8977-03c7017bf5c9/kuadrant-console-plugin/0.log" Jul 02 20:59:18.632414 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:18.632374 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d957c959b-4767p_642c49dc-9648-4b04-a6ba-b775546004ac/kube-auth-proxy/0.log" Jul 02 20:59:19.102359 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:19.102328 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-l6qqw_e77a61db-6c15-41dd-8d9b-a1e8aac0d648/payload-processing/0.log" Jul 02 20:59:19.677473 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:19.677442 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-krgm2_43463c81-11aa-4a42-bfb0-b55064428a4c/main/0.log" Jul 02 20:59:20.007831 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:20.007745 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-t6h5q_d8fce5e7-9a3e-4a14-a868-ffedc13b976e/main/0.log" Jul 02 20:59:20.121666 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:20.121622 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-mq6v5_a6cd414b-430d-471f-9a9e-57f98988d182/main/0.log" Jul 02 20:59:24.162227 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:24.162184 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:24.259707 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:24.259674 2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" limit=1024 Jul 02 20:59:27.134817 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:27.134785 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nkzfc_81dfeaf2-e4c7-4ac8-8ec3-7b4bb8ed5eee/global-pull-secret-syncer/0.log" Jul 02 20:59:27.183421 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:27.183381 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hd7jg_3aae5d08-0b74-4611-8932-d1afc7b80412/konnectivity-agent/0.log" Jul 02 20:59:27.263778 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:27.263752 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-15.ec2.internal_99ae426d085798cd8010de0c176c4766/haproxy/0.log" Jul 02 20:59:32.497267 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:32.497238 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-d9hxp_d6b2d93d-1f86-4f70-8977-03c7017bf5c9/kuadrant-console-plugin/0.log" Jul 02 20:59:34.628798 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:34.628765 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-mqr6k_a7e7fea8-aaec-47ae-9137-5be0b93671bb/cluster-monitoring-operator/0.log" Jul 02 20:59:34.661673 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:34.661649 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-q24zq_0cff93e5-3bd8-4d6d-a05d-3649be4e11a5/kube-state-metrics/0.log" Jul 02 20:59:34.682971 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:34.682944 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-q24zq_0cff93e5-3bd8-4d6d-a05d-3649be4e11a5/kube-rbac-proxy-main/0.log" Jul 02 20:59:34.704157 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:34.704125 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-q24zq_0cff93e5-3bd8-4d6d-a05d-3649be4e11a5/kube-rbac-proxy-self/0.log" Jul 02 20:59:34.908379 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:34.908347 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9qvf_b8b6f017-988d-4d7f-930a-aaf71f52a7f2/node-exporter/0.log" Jul 02 20:59:34.942500 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:34.942464 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9qvf_b8b6f017-988d-4d7f-930a-aaf71f52a7f2/kube-rbac-proxy/0.log" Jul 02 20:59:34.968972 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:34.968950 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r9qvf_b8b6f017-988d-4d7f-930a-aaf71f52a7f2/init-textfile/0.log" Jul 02 20:59:35.073715 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:35.073688 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-shzp9_711c2aad-274f-4c23-8280-9c9aebefec56/kube-rbac-proxy-main/0.log" Jul 02 20:59:35.094758 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:35.094714 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-shzp9_711c2aad-274f-4c23-8280-9c9aebefec56/kube-rbac-proxy-self/0.log" Jul 02 20:59:35.116910 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:35.116883 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-shzp9_711c2aad-274f-4c23-8280-9c9aebefec56/openshift-state-metrics/0.log" Jul 02 20:59:35.366403 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:35.366329 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-vh5g2_57e179a8-99ec-42a4-875e-784a933b3c82/prometheus-operator/0.log" Jul 02 20:59:35.387293 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:35.387266 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-vh5g2_57e179a8-99ec-42a4-875e-784a933b3c82/kube-rbac-proxy/0.log" Jul 02 20:59:35.468626 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:35.468581 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cdfd6464f-wg729_2bb49ec6-6170-441f-8698-180685fd280d/telemeter-client/0.log" Jul 02 20:59:35.491093 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:35.491065 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cdfd6464f-wg729_2bb49ec6-6170-441f-8698-180685fd280d/reload/0.log" Jul 02 20:59:35.512040 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:35.512016 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6cdfd6464f-wg729_2bb49ec6-6170-441f-8698-180685fd280d/kube-rbac-proxy/0.log" Jul 02 20:59:37.455195 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.455165 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:59:37.459298 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.459279 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/3.log" Jul 02 20:59:37.722746 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.722663 2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn"] Jul 02 20:59:37.723049 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.723035 2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781b5a50-4bb8-4bb5-8b21-0727acdf306b" containerName="test-curl-6908-140273651856368" Jul 02 20:59:37.723098 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.723051 2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="781b5a50-4bb8-4bb5-8b21-0727acdf306b" containerName="test-curl-6908-140273651856368" Jul 02 20:59:37.723133 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.723124 2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="781b5a50-4bb8-4bb5-8b21-0727acdf306b" containerName="test-curl-6908-140273651856368" Jul 02 20:59:37.726456 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.726436 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.728968 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.728944 2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9mtnv\"/\"default-dockercfg-5s2gd\"" Jul 02 20:59:37.729067 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.728970 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9mtnv\"/\"kube-root-ca.crt\"" Jul 02 20:59:37.729067 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.729015 2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9mtnv\"/\"openshift-service-ca.crt\"" Jul 02 20:59:37.737091 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.737066 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn"] Jul 02 20:59:37.874497 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.874459 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-lib-modules\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.874497 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.874499 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-podres\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.874740 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.874604 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24zrp\" (UniqueName: \"kubernetes.io/projected/2176631e-6be4-4614-bd4d-af7a07e9c1bb-kube-api-access-24zrp\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.874740 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.874667 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-sys\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.874825 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.874738 2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-proc\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.976125 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.976040 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-proc\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.976125 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.976087 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-lib-modules\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.976125 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.976104 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-podres\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.976381 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.976143 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24zrp\" (UniqueName: \"kubernetes.io/projected/2176631e-6be4-4614-bd4d-af7a07e9c1bb-kube-api-access-24zrp\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.976381 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.976180 2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-sys\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.976381 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.976187 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-proc\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.976381 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.976269 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-lib-modules\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.976381 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.976289 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-sys\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.976381 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.976308 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2176631e-6be4-4614-bd4d-af7a07e9c1bb-podres\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.984711 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.984692 2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24zrp\" (UniqueName: \"kubernetes.io/projected/2176631e-6be4-4614-bd4d-af7a07e9c1bb-kube-api-access-24zrp\") pod \"perf-node-gather-daemonset-ps7qn\" (UID: \"2176631e-6be4-4614-bd4d-af7a07e9c1bb\") " pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:37.998486 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:37.998465 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5db4d66dfc-n4snw_55bbf3ab-8c87-44ca-89cd-d942b106e12c/console/0.log" Jul 02 20:59:38.033288 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:38.033263 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-h6w5f_0ff452e7-4b5f-4e94-84b6-245ce46bb584/download-server/0.log" Jul 02 20:59:38.037250 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:38.037233 2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:38.177080 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:38.177024 2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn"] Jul 02 20:59:38.179507 ip-10-0-132-15 kubenswrapper[2565]: W0702 20:59:38.179481 2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2176631e_6be4_4614_bd4d_af7a07e9c1bb.slice/crio-e64776b278a97cdd2eb456615b3c814a85c550d936d1adcf4910b546fd8c8650 WatchSource:0}: Error finding container e64776b278a97cdd2eb456615b3c814a85c550d936d1adcf4910b546fd8c8650: Status 404 returned error can't find the container with id e64776b278a97cdd2eb456615b3c814a85c550d936d1adcf4910b546fd8c8650 Jul 02 20:59:38.827145 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:38.827108 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" event={"ID":"2176631e-6be4-4614-bd4d-af7a07e9c1bb","Type":"ContainerStarted","Data":"6e7d5bbfcd671e75dfe0187143e5323d8dda5352fe998d95427766c0248ae4f0"} Jul 02 20:59:38.827544 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:38.827168 2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" event={"ID":"2176631e-6be4-4614-bd4d-af7a07e9c1bb","Type":"ContainerStarted","Data":"e64776b278a97cdd2eb456615b3c814a85c550d936d1adcf4910b546fd8c8650"} Jul 02 20:59:38.827544 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:38.827234 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:38.847017 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:38.846954 2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" podStartSLOduration=1.8469316569999998 podStartE2EDuration="1.846931657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 20:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 20:59:38.843823719 +0000 UTC m=+2687.840126954" watchObservedRunningTime="2026-07-02 20:59:38.846931657 +0000 UTC m=+2687.843234890" Jul 02 20:59:39.727179 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:39.727152 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmr7z_3386b537-14c7-4ed3-9c7c-7d5d83e7ae93/dns/0.log" Jul 02 20:59:39.751298 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:39.751271 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hmr7z_3386b537-14c7-4ed3-9c7c-7d5d83e7ae93/kube-rbac-proxy/0.log" Jul 02 20:59:39.924102 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:39.924069 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8thwh_d462c3b4-ffb3-4449-aff4-cd348fbc835c/dns-node-resolver/0.log" Jul 02 20:59:40.726274 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:40.726243 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jz4g9_a9ef4988-0ad9-491c-9935-929c5a0eb8ba/node-ca/0.log" Jul 02 20:59:41.896517 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:41.896445 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d957c959b-4767p_642c49dc-9648-4b04-a6ba-b775546004ac/kube-auth-proxy/0.log" Jul 02 20:59:42.091047 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:42.091017 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-l6qqw_e77a61db-6c15-41dd-8d9b-a1e8aac0d648/payload-processing/0.log" Jul 02 20:59:42.801182 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:42.801158 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gj22w_efc20391-221b-452a-8960-bd8d55818dc9/serve-healthcheck-canary/0.log" Jul 02 20:59:43.310759 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:43.310734 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-7r8wz_6e15797b-e921-4425-b53f-94b801477686/insights-operator/0.log" Jul 02 20:59:43.311133 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:43.311101 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-7r8wz_6e15797b-e921-4425-b53f-94b801477686/insights-operator/1.log" Jul 02 20:59:43.344724 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:43.344700 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5wv7t_dbe0c436-b6d1-4773-9982-0d0323b30552/kube-rbac-proxy/0.log" Jul 02 20:59:43.372849 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:43.372829 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5wv7t_dbe0c436-b6d1-4773-9982-0d0323b30552/exporter/0.log" Jul 02 20:59:43.397468 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:43.397450 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5wv7t_dbe0c436-b6d1-4773-9982-0d0323b30552/extractor/0.log" Jul 02 20:59:44.843659 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:44.843632 2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9mtnv/perf-node-gather-daemonset-ps7qn" Jul 02 20:59:45.921481 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:45.921451 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-55f547546d-456dr_5a7c2b51-b00f-41e8-92d3-d97e2e13d53e/maas-api/0.log" Jul 02 20:59:46.196580 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:46.196499 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b478dcdf-4p54h_3929840f-933b-4207-8684-5a10e32135c0/manager/0.log" Jul 02 20:59:46.219916 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:46.219888 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-78b478dcdf-4p54h_3929840f-933b-4207-8684-5a10e32135c0/copy-manifests/0.log" Jul 02 20:59:46.431285 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:46.431258 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-h9t5q_98c73987-16d5-4082-bff8-ebfbf34c0406/postgres/0.log" Jul 02 20:59:48.058289 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:48.058257 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5b47c96757-z8lzx_60f6cb36-921e-4f3e-8598-994e01fab8c0/manager/0.log" Jul 02 20:59:48.071110 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:48.071088 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d957c959b-4767p_642c49dc-9648-4b04-a6ba-b775546004ac/kube-auth-proxy/0.log" Jul 02 20:59:48.082427 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:48.082398 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-7fvt6_c28875e4-241d-4f91-a517-83d78a5c4dd6/openshift-lws-operator/0.log" Jul 02 20:59:48.206761 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:48.206732 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-l6qqw_e77a61db-6c15-41dd-8d9b-a1e8aac0d648/payload-processing/0.log" Jul 02 20:59:49.524760 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:49.524722 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-d957c959b-4767p_642c49dc-9648-4b04-a6ba-b775546004ac/kube-auth-proxy/0.log" Jul 02 20:59:49.674263 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:49.674232 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-l6qqw_e77a61db-6c15-41dd-8d9b-a1e8aac0d648/payload-processing/0.log" Jul 02 20:59:51.661763 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:51.661719 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:59:51.669180 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:51.669156 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-7b8w7_5ba6abb2-3ddd-4296-9858-6de372075f3a/console-operator/2.log" Jul 02 20:59:54.501303 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:54.501272 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c7nw_fa235a05-22fe-4e3a-adc5-705913aad0c4/kube-multus-additional-cni-plugins/0.log" Jul 02 20:59:54.524557 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:54.524531 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c7nw_fa235a05-22fe-4e3a-adc5-705913aad0c4/egress-router-binary-copy/0.log" Jul 02 20:59:54.546467 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:54.546438 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c7nw_fa235a05-22fe-4e3a-adc5-705913aad0c4/cni-plugins/0.log" Jul 02 20:59:54.566661 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:54.566635 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c7nw_fa235a05-22fe-4e3a-adc5-705913aad0c4/bond-cni-plugin/0.log" Jul 02 20:59:54.587924 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:54.587897 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c7nw_fa235a05-22fe-4e3a-adc5-705913aad0c4/routeoverride-cni/0.log" Jul 02 20:59:54.613823 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:54.613803 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c7nw_fa235a05-22fe-4e3a-adc5-705913aad0c4/whereabouts-cni-bincopy/0.log" Jul 02 20:59:54.636764 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:54.636741 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5c7nw_fa235a05-22fe-4e3a-adc5-705913aad0c4/whereabouts-cni/0.log" Jul 02 20:59:55.006779 ip-10-0-132-15 kubenswrapper[2565]: I0702 20:59:55.006753 2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nswz8_0b7cafcc-8972-4475-8f86-4e22f41d1604/kube-multus/0.log"