Jul 10 22:20:04.845128 ip-10-0-140-116 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 22:20:04.845140 ip-10-0-140-116 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 22:20:04.845149 ip-10-0-140-116 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 22:20:04.845452 ip-10-0-140-116 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 22:20:16.467871 ip-10-0-140-116 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 22:20:16.467890 ip-10-0-140-116 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 8a182a9754a74503be0229a23e3303f9 -- Jul 10 22:21:52.021729 ip-10-0-140-116 systemd[1]: Starting Kubernetes Kubelet... Jul 10 22:21:52.486251 ip-10-0-140-116 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 22:21:52.486251 ip-10-0-140-116 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 22:21:52.486251 ip-10-0-140-116 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 22:21:52.486251 ip-10-0-140-116 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 22:21:52.486251 ip-10-0-140-116 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 22:21:52.486251 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.483644 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487357 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487367 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487371 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487374 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487377 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487379 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487382 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487384 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487387 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487390 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487393 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487395 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487398 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487401 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487403 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487406 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487409 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487411 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487414 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:21:52.502570 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487417 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487420 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487422 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487427 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487430 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487433 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487449 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487452 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487455 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487458 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487468 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487471 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487474 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487477 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487481 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487484 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487486 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487489 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487491 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487495 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:21:52.517141 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487497 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487500 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487502 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487505 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487507 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487509 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487512 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487514 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487518 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487522 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487524 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487527 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487530 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487532 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487535 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487538 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487540 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487543 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487546 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:21:52.518923 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487548 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487551 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487553 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487555 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487558 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487561 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487563 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487566 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487569 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487571 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487573 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487576 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487579 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487582 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487584 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487586 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487590 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487592 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487594 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487597 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:21:52.521373 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487599 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487602 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487605 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487607 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487610 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487612 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487614 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487617 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487619 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487622 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487625 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487627 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487630 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487632 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487636 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487640 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487643 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487645 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487648 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:21:52.528777 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487650 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.487653 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488363 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488369 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488373 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488376 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488380 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488383 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488386 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488389 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488392 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488395 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488398 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488401 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488404 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488407 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488411 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488413 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488416 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488419 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:21:52.530993 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488422 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488424 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488427 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488430 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488433 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488451 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488455 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488458 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488460 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488463 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488466 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488468 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488471 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488473 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488476 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488478 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488482 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488485 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488487 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488492 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:21:52.532856 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488495 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488499 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488501 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488504 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488507 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488509 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488512 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488515 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488517 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488520 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488523 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488527 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488530 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488533 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488536 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488538 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488541 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488543 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488546 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:21:52.535170 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488548 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488551 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488553 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488556 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488558 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488561 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488563 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488567 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488570 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488572 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488575 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488577 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488582 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488585 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488587 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488590 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488594 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488597 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488601 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:21:52.548583 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488603 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488606 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488609 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488612 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488616 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488619 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488621 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488625 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488629 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488631 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488634 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488637 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488640 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488643 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488645 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488648 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488650 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488653 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488655 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488658 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:21:52.550763 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488661 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488663 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.488666 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489038 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489043 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489047 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489049 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489052 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489054 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489057 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489059 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489061 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489064 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489066 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489069 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489071 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489074 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489077 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489081 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:21:52.552523 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489084 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489087 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489090 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489093 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489095 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489098 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489100 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489103 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489105 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489107 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489110 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489112 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489114 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489117 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489119 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489121 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489124 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489127 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489131 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489133 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:21:52.554931 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489136 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489138 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489141 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489143 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489146 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489148 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489151 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489153 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489156 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489159 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489162 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489164 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489167 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489169 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489172 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489174 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489177 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489179 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489182 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489184 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:21:52.624727 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489187 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:21:52.560864 ip-10-0-140-116 systemd[1]: Started Kubernetes Kubelet. Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489190 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489192 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489195 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489197 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489199 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489202 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489204 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489207 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489209 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489213 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489216 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489218 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489220 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489223 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489225 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489228 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489230 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489233 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489235 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:21:52.775773 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489237 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489240 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489242 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489245 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489248 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489252 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489256 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489258 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489261 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489263 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489266 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489269 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489272 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489274 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489276 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489279 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489282 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489285 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489287 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:21:52.777533 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489290 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489292 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489294 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489299 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489364 2559 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489370 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489376 2559 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489380 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489385 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489388 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489393 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489397 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489400 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489403 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489406 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489409 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489412 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489415 2559 flags.go:64] FLAG: --cgroup-root="" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489418 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489421 2559 flags.go:64] FLAG: --client-ca-file="" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489423 2559 flags.go:64] FLAG: --cloud-provider="external" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489426 2559 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489430 2559 flags.go:64] FLAG: --cluster-domain="" Jul 10 22:21:52.778173 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489432 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489454 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489458 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489461 2559 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489464 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489467 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489470 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489473 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489476 2559 flags.go:64] FLAG: --contention-profiling="false" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489479 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489482 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489485 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489489 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489493 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489496 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489498 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489501 2559 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489505 2559 flags.go:64] FLAG: --enable-server="true" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489508 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489512 2559 flags.go:64] FLAG: --event-burst="100" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489515 2559 flags.go:64] FLAG: --event-qps="50" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489518 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489521 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489524 2559 flags.go:64] FLAG: --eviction-hard="" Jul 10 22:21:52.778846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489527 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489530 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489533 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489536 2559 flags.go:64] FLAG: --eviction-soft="" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489539 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489542 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489544 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489547 2559 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489550 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489553 2559 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489555 2559 flags.go:64] FLAG: --feature-gates="" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489559 2559 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489562 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489565 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489568 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489570 2559 flags.go:64] FLAG: --healthz-port="10248" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489573 2559 flags.go:64] FLAG: --help="false" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489576 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-140-116.ec2.internal" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489579 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489582 2559 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489586 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489589 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489593 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489596 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 22:21:52.779552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489598 2559 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489602 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489605 2559 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489608 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489611 2559 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489613 2559 flags.go:64] FLAG: --kube-reserved="" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489616 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489618 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489621 2559 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489624 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489627 2559 flags.go:64] FLAG: --lock-file="" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489629 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489634 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489636 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489642 2559 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489644 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489647 2559 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489649 2559 flags.go:64] FLAG: --logging-format="text" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489652 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489655 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489658 2559 flags.go:64] FLAG: --manifest-url="" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489661 2559 flags.go:64] FLAG: --manifest-url-header="" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489665 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489667 2559 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489671 2559 flags.go:64] FLAG: --max-pods="110" Jul 10 22:21:52.780319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489674 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489677 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489679 2559 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489683 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489686 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489689 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489692 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489698 2559 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489702 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489705 2559 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489708 2559 flags.go:64] FLAG: --pod-cidr="" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489711 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489716 2559 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489718 2559 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489721 2559 flags.go:64] FLAG: --pods-per-core="0" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489724 2559 flags.go:64] FLAG: --port="10250" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489727 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489730 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-068b1f7eb272a31ea" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489733 2559 flags.go:64] FLAG: --qos-reserved="" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489735 2559 flags.go:64] FLAG: --read-only-port="10255" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489739 2559 flags.go:64] FLAG: --register-node="true" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489741 2559 flags.go:64] FLAG: --register-with-taints="" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489745 2559 flags.go:64] FLAG: --registry-burst="10" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489747 2559 flags.go:64] FLAG: --registry-qps="5" Jul 10 22:21:52.781143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489750 2559 flags.go:64] FLAG: --reserved-cpus="" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489753 2559 flags.go:64] FLAG: --reserved-memory="" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489757 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489759 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489762 2559 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489765 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489767 2559 flags.go:64] FLAG: --runonce="false" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489770 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489773 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489776 2559 flags.go:64] FLAG: --seccomp-default="false" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489779 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489781 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489785 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489788 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489791 2559 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489794 2559 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489798 2559 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489801 2559 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489803 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489806 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489809 2559 flags.go:64] FLAG: --system-cgroups="" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489811 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489816 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489819 2559 flags.go:64] FLAG: --tls-cert-file="" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489821 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 22:21:52.782123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489825 2559 flags.go:64] FLAG: --tls-min-version="" Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489828 2559 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489830 2559 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489833 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489837 2559 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489839 2559 flags.go:64] FLAG: --v="2" Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489843 2559 flags.go:64] FLAG: --version="false" Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489847 2559 flags.go:64] FLAG: --vmodule="" Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489856 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.489859 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489974 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489977 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489981 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489984 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489986 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489989 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489992 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489995 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.489997 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490001 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490003 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490006 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:21:52.783256 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490009 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490013 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490015 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490018 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490020 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490023 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490025 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490029 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490032 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490035 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490038 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490040 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490043 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490045 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490048 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490056 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490059 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490062 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490064 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490066 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:21:52.784025 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490069 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490071 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490074 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490076 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490079 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490081 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490083 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490086 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490088 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490092 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490095 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490098 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490101 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490107 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490109 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490112 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490115 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490117 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490120 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:21:52.784685 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490122 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490125 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490127 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490129 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490132 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490134 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490137 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490139 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490141 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490149 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490152 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490154 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490158 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490161 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490164 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490167 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490170 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490172 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490175 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:21:52.785303 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490177 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490180 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490182 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490186 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490188 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490191 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490193 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490197 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490200 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490202 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490205 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490207 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490210 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490212 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490215 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490217 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490220 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490222 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490225 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:21:52.785898 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490227 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490230 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490232 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490235 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490243 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490245 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490247 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490250 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490252 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.490255 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.491115 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.498716 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.498732 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.498750 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.498755 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 22:21:52.786398 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498817 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498822 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498826 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498829 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498831 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498834 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498837 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498839 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498842 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498844 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498847 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498849 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498852 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498855 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498858 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498861 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498863 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498866 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498868 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:21:52.786900 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498871 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498875 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498879 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498882 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498885 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498888 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498891 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498894 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498896 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498899 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498901 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498904 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498906 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498910 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498913 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498916 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498919 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498922 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498924 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:21:52.939568 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498927 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498930 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498933 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498935 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498938 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498941 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498944 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498946 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498951 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498953 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498956 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498958 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498961 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498964 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498967 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498969 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498972 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498975 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498977 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498980 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:21:52.940285 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498982 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498986 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498988 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498990 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498993 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498995 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.498998 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499000 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499002 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499005 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499007 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499010 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499012 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499015 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499017 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499020 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499023 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499026 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499028 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499031 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:21:52.940884 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499034 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499037 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499039 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499042 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499044 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499047 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499050 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499052 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499056 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499060 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499064 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499067 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499069 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499072 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499074 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499077 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499079 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499083 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499085 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499088 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:21:52.941463 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499091 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.499096 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499219 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499223 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499226 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499229 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499232 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499235 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499237 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499240 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499242 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499245 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499249 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499251 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499254 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 22:21:52.941971 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499256 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499259 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499261 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499264 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499267 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499269 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499271 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499274 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499277 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499279 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499282 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499284 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499287 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499289 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499292 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499295 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499298 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499300 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499302 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 22:21:52.942403 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499305 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499308 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499310 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499313 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499316 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499319 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499322 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499324 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499327 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499329 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499333 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499336 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499338 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499341 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499344 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499348 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499351 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499354 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499357 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 22:21:52.943180 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499360 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499362 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499366 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499368 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499371 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499373 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499375 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499378 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499380 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499385 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499387 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499389 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499392 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499394 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499397 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499399 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499402 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499404 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499407 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499409 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 22:21:52.943797 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499412 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499414 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499416 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499421 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499424 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499426 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499428 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499431 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499433 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499454 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499458 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499460 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499463 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499465 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499468 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499471 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499474 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499476 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499479 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499481 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 22:21:53.148074 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499484 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499488 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499490 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499493 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499495 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499497 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499500 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:52.499502 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.499507 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.499607 2559 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.501564 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.502994 2559 server.go:1019] "Starting client certificate rotation" Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.503097 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.503138 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 22:21:53.148770 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.505763 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 22:21:53.149177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.506850 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 22:21:53.149177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.522822 2559 log.go:25] "Validated CRI v1 runtime API" Jul 10 22:21:53.149177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.525918 2559 log.go:25] "Validated CRI v1 image API" Jul 10 22:21:53.149177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.526751 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 22:21:53.149177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.529492 2559 fs.go:135] Filesystem UUIDs: map[373097ff-0d86-4b5b-9260-d9cec5e9c19f:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 945e0d2c-2350-481c-bda9-21a9f0da1cbe:/dev/nvme0n1p4] Jul 10 22:21:53.149177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.529511 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 22:21:53.149177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.534837 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.534950 2559 manager.go:216] Machine: {Timestamp:2026-07-10 22:21:52.534415083 +0000 UTC m=+0.382696954 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3179267 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ef50a6970888b2a0fc2ff4efba950 SystemUUID:ec2ef50a-6970-888b-2a0f-c2ff4efba950 BootID:8a182a97-54a7-4503-be02-29a23e3303f9 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f3:6a:6b:55:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f3:6a:6b:55:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:91:67:de:f9:83 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.535057 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.535160 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.538762 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.538784 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-116.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.539661 2559 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.539671 2559 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.539678 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.539688 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.541376 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.545382 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.545520 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.545550 2559 kubelet.go:486] "Attempting to sync node with API server" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.545561 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.545576 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.545585 2559 kubelet.go:398] "Adding apiserver pod source" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.545594 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 22:21:53.149607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.545921 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.545935 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.548731 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.555178 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.555221 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.555433 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.555476 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.555487 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.555496 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.555508 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.555517 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.556108 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.556118 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.556124 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.556130 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.556147 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.556156 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.556422 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.556509 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.557013 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.557021 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.560001 2559 server.go:1268] "Started kubelet" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.560092 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.560116 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.560170 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.561224 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.564981 2559 server.go:310] "Adding debug handlers to kubelet server" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.565715 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.569236 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.569132 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-116.ec2.internal.18c10dcf91257665 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-116.ec2.internal,UID:ip-10-0-140-116.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-116.ec2.internal,},FirstTimestamp:2026-07-10 22:21:52.559969893 +0000 UTC m=+0.408251759,LastTimestamp:2026-07-10 22:21:52.559969893 +0000 UTC m=+0.408251759,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-116.ec2.internal,}" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.570729 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-116.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.570858 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.571144 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.571813 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.571812 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.571922 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.572062 2559 reconstruct.go:97] "Volume reconstruction finished" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.572084 2559 reconciler.go:29] "Reconciler: start to sync state" Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.572226 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.572242 2559 factory.go:55] Registering systemd factory Jul 10 22:21:53.150177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.572251 2559 factory.go:223] Registration of the systemd container factory successfully Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.572482 2559 factory.go:153] Registering CRI-O factory Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.572495 2559 factory.go:223] Registration of the crio container factory successfully Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.572513 2559 factory.go:103] Registering Raw factory Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.572522 2559 manager.go:1196] Started watching for new ooms in manager Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.572952 2559 manager.go:349] Starting recovery of all containers Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.573591 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.573664 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-116.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.582070 2559 manager.go:354] Recovery completed Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.583382 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j6fbs" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.587301 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.589328 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.589356 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.589371 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.590049 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.590057 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.590070 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.590064 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.590342 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-j6fbs" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.590866 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.590884 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.590902 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.591381 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-116.ec2.internal.18c10dcf92e59fec default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-116.ec2.internal,UID:ip-10-0-140-116.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-116.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-116.ec2.internal,},FirstTimestamp:2026-07-10 22:21:52.589340652 +0000 UTC m=+0.437622524,LastTimestamp:2026-07-10 22:21:52.589340652 +0000 UTC m=+0.437622524,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-116.ec2.internal,}" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.592275 2559 policy_none.go:49] "None policy: Start" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.592287 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.592297 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.594374 2559 policy_none.go:47] "Start" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.660970 2559 manager.go:337] "Starting Device Plugin manager" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.661126 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.661138 2559 server.go:87] "Starting device plugin registration server" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.661366 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.661375 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.661505 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.661606 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.661617 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.662431 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.662489 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.718005 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.719092 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.719128 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.719152 2559 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 22:21:53.151250 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.719181 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.721583 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.762467 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.763270 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.763298 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.763312 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.763343 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.775336 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.775355 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-116.ec2.internal\": node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.798431 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.819502 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-116.ec2.internal"] Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.819557 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.820296 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.820325 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.820338 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.822682 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.822809 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.822836 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.824446 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.824466 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.824475 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.824503 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.824521 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.824533 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.826707 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.826734 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.827930 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientMemory" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.827959 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.827976 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeHasSufficientPID" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.857114 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-116.ec2.internal\" not found" node="ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.861336 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-116.ec2.internal\" not found" node="ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.874061 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/419dafcf2fcd506b2efc42482d8111c7-config\") pod \"kube-apiserver-proxy-ip-10-0-140-116.ec2.internal\" (UID: \"419dafcf2fcd506b2efc42482d8111c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.874082 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f08f323b708e7c395ca2dce63145ea16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal\" (UID: \"f08f323b708e7c395ca2dce63145ea16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.154660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.874098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f08f323b708e7c395ca2dce63145ea16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal\" (UID: \"f08f323b708e7c395ca2dce63145ea16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.155517 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.899214 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.155517 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.974312 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f08f323b708e7c395ca2dce63145ea16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal\" (UID: \"f08f323b708e7c395ca2dce63145ea16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.155517 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.974406 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f08f323b708e7c395ca2dce63145ea16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal\" (UID: \"f08f323b708e7c395ca2dce63145ea16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.155517 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.974426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/419dafcf2fcd506b2efc42482d8111c7-config\") pod \"kube-apiserver-proxy-ip-10-0-140-116.ec2.internal\" (UID: \"419dafcf2fcd506b2efc42482d8111c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.155517 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.974363 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f08f323b708e7c395ca2dce63145ea16-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal\" (UID: \"f08f323b708e7c395ca2dce63145ea16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.155517 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.974477 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/419dafcf2fcd506b2efc42482d8111c7-config\") pod \"kube-apiserver-proxy-ip-10-0-140-116.ec2.internal\" (UID: \"419dafcf2fcd506b2efc42482d8111c7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.155517 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:52.974519 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f08f323b708e7c395ca2dce63145ea16-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal\" (UID: \"f08f323b708e7c395ca2dce63145ea16\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.155517 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:52.999407 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.155517 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:53.100153 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.337794 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:53.159324 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.337794 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:53.164386 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-116.ec2.internal" Jul 10 22:21:53.337794 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:53.201226 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.337794 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:53.301724 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.563060 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:53.402172 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.563060 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:53.456066 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:21:53.563060 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:53.503315 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.563060 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:53.503336 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 22:21:53.563060 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:53.503440 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 22:21:53.563060 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:53.503491 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 22:21:53.787409 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:53.593964 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 22:16:52 +0000 UTC" deadline="2028-04-12 01:16:57.945033492 +0000 UTC" Jul 10 22:21:53.787409 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:53.593989 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15386h55m4.351047286s" Jul 10 22:21:53.787409 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:53.604137 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:53.787409 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:53.704745 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:54.031675 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:53.805792 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:54.031675 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:53.856761 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08f323b708e7c395ca2dce63145ea16.slice/crio-6bfcc661287cfebe7338212ec8cf0da23b85cc67ec9fcafc642d230f79cde32e WatchSource:0}: Error finding container 6bfcc661287cfebe7338212ec8cf0da23b85cc67ec9fcafc642d230f79cde32e: Status 404 returned error can't find the container with id 6bfcc661287cfebe7338212ec8cf0da23b85cc67ec9fcafc642d230f79cde32e Jul 10 22:21:54.031675 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:53.857389 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419dafcf2fcd506b2efc42482d8111c7.slice/crio-1327b4fb6102dc0589c76ca36322b4a4f2c29d226df2171bba646e90ab2f5b08 WatchSource:0}: Error finding container 1327b4fb6102dc0589c76ca36322b4a4f2c29d226df2171bba646e90ab2f5b08: Status 404 returned error can't find the container with id 1327b4fb6102dc0589c76ca36322b4a4f2c29d226df2171bba646e90ab2f5b08 Jul 10 22:21:54.031675 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:53.862104 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:21:54.031675 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:53.887902 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:21:54.031675 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:53.906025 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-116.ec2.internal\" not found" Jul 10 22:21:54.031675 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.002055 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:21:54.287143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.072476 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-116.ec2.internal" Jul 10 22:21:54.287143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.080622 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 22:21:54.287143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.081761 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" Jul 10 22:21:54.287143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.093044 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.547105 2559 apiserver.go:52] "Watching apiserver" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.559799 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.560286 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-9t2tv","kube-system/kube-apiserver-proxy-ip-10-0-140-116.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz","openshift-dns/node-resolver-m5kxh","openshift-network-operator/iptables-alerter-bzzg9","openshift-ovn-kubernetes/ovnkube-node-tkx48","kube-system/global-pull-secret-syncer-krhr5","openshift-cluster-node-tuning-operator/tuned-j97hm","openshift-image-registry/node-ca-7hj6f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal","openshift-multus/multus-additional-cni-plugins-nfvl2","openshift-multus/multus-fmj2g","openshift-multus/network-metrics-daemon-wbxsz","openshift-network-diagnostics/network-check-target-98cjk"] Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.563090 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.565930 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.566274 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.566390 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5frjd\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.566973 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:54.567059 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.571002 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:54.571758 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.572002 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m5kxh" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.572594 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.574151 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bzzg9" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.574719 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qwjv2\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.574725 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.574968 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.576447 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.576772 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v6gqb\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.576985 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.577060 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.577231 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.579914 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.580097 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.580447 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.580893 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rck64\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.580900 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.580999 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.581311 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.581506 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 22:21:54.588874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.582358 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.582877 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/976133c2-7307-4130-a56d-56f9727c7ba6-dbus\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.582891 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.582912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7-tmp-dir\") pod \"node-resolver-m5kxh\" (UID: \"2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7\") " pod="openshift-dns/node-resolver-m5kxh" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.582941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e625366f-c6d4-482e-88f8-de3023a513f6-host-slash\") pod \"iptables-alerter-bzzg9\" (UID: \"e625366f-c6d4-482e-88f8-de3023a513f6\") " pod="openshift-network-operator/iptables-alerter-bzzg9" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.582967 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dchj\" (UniqueName: \"kubernetes.io/projected/e625366f-c6d4-482e-88f8-de3023a513f6-kube-api-access-6dchj\") pod \"iptables-alerter-bzzg9\" (UID: \"e625366f-c6d4-482e-88f8-de3023a513f6\") " pod="openshift-network-operator/iptables-alerter-bzzg9" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.582992 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e7057db-a92a-460c-9cbc-7f9815afa729-konnectivity-ca\") pod \"konnectivity-agent-9t2tv\" (UID: \"8e7057db-a92a-460c-9cbc-7f9815afa729\") " pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.583017 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znqhg\" (UniqueName: \"kubernetes.io/projected/2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7-kube-api-access-znqhg\") pod \"node-resolver-m5kxh\" (UID: \"2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7\") " pod="openshift-dns/node-resolver-m5kxh" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.583069 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.583076 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbw4m\" (UniqueName: \"kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m\") pod \"network-check-target-98cjk\" (UID: \"43f8635d-832b-42d5-932b-5e06411e985a\") " pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.583106 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/976133c2-7307-4130-a56d-56f9727c7ba6-kubelet-config\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.583134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.583149 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.583163 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e7057db-a92a-460c-9cbc-7f9815afa729-agent-certs\") pod \"konnectivity-agent-9t2tv\" (UID: \"8e7057db-a92a-460c-9cbc-7f9815afa729\") " pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.583188 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7-hosts-file\") pod \"node-resolver-m5kxh\" (UID: \"2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7\") " pod="openshift-dns/node-resolver-m5kxh" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.583212 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e625366f-c6d4-482e-88f8-de3023a513f6-iptables-alerter-script\") pod \"iptables-alerter-bzzg9\" (UID: \"e625366f-c6d4-482e-88f8-de3023a513f6\") " pod="openshift-network-operator/iptables-alerter-bzzg9" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.583315 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-skc8r\"" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.584292 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.584921 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.585573 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-dcr6m\"" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.585789 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7hj6f" Jul 10 22:21:54.592091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.587502 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 22:21:54.593998 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.588331 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.593998 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.589423 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 22:21:54.593998 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.590313 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 22:21:54.593998 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.590700 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 22:21:54.593998 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.590886 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 22:21:54.593998 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.591251 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 22:21:54.593998 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.593882 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rmg5q\"" Jul 10 22:21:54.595874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.594001 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.595874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.594136 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 22:21:54.595874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.594913 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ctbdd\"" Jul 10 22:21:54.595874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.595071 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 22:21:54.595874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.595369 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 22:21:54.596854 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.595993 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:54.596854 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:54.596079 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:21:54.598898 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.596918 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 22:21:54.598898 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.596966 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jmgpx\"" Jul 10 22:21:54.614805 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.614147 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7pmxq" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.623381 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7pmxq" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.630869 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.672923 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683475 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fec2b764-2853-4ff8-95ff-b0264fb8e16e-ovn-node-metrics-cert\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683512 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-daemon-config\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fec2b764-2853-4ff8-95ff-b0264fb8e16e-ovnkube-script-lib\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683571 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-run-openvswitch\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683619 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-cni-netd\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-sys-fs\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683671 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c6ad821-e27e-49a2-85ec-354c19403be5-cni-binary-copy\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-cni-bin\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683719 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-run-k8s-cni-cncf-io\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683743 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-systemd\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dchj\" (UniqueName: \"kubernetes.io/projected/e625366f-c6d4-482e-88f8-de3023a513f6-kube-api-access-6dchj\") pod \"iptables-alerter-bzzg9\" (UID: \"e625366f-c6d4-482e-88f8-de3023a513f6\") " pod="openshift-network-operator/iptables-alerter-bzzg9" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683808 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-node-log\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683831 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-socket-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.694486 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683878 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6171ceb6-38d3-46f3-a504-42094bd28cde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.683941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-slash\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:54.684008 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684057 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-run-multus-certs\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:54.684089 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret podName:976133c2-7307-4130-a56d-56f9727c7ba6 nodeName:}" failed. No retries permitted until 2026-07-10 22:21:55.184064371 +0000 UTC m=+3.032346251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret") pod "global-pull-secret-syncer-krhr5" (UID: "976133c2-7307-4130-a56d-56f9727c7ba6") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684125 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-sysctl-d\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684167 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g26jp\" (UniqueName: \"kubernetes.io/projected/0045faa6-bed2-460c-b8f7-eaf7f30a7685-kube-api-access-g26jp\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d1f9ea3-e113-4617-bd09-fc1b793a2bbe-host\") pod \"node-ca-7hj6f\" (UID: \"1d1f9ea3-e113-4617-bd09-fc1b793a2bbe\") " pod="openshift-image-registry/node-ca-7hj6f" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684216 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d1f9ea3-e113-4617-bd09-fc1b793a2bbe-serviceca\") pod \"node-ca-7hj6f\" (UID: \"1d1f9ea3-e113-4617-bd09-fc1b793a2bbe\") " pod="openshift-image-registry/node-ca-7hj6f" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684241 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-kubelet\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684266 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fec2b764-2853-4ff8-95ff-b0264fb8e16e-ovnkube-config\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684292 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpnhz\" (UniqueName: \"kubernetes.io/projected/54dafdf9-e230-4a2f-9f33-df68421e3c5a-kube-api-access-qpnhz\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684317 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-host\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684341 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-cnibin\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684385 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdp72\" (UniqueName: \"kubernetes.io/projected/6171ceb6-38d3-46f3-a504-42094bd28cde-kube-api-access-jdp72\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:54.695893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684435 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-cnibin\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684465 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e7057db-a92a-460c-9cbc-7f9815afa729-agent-certs\") pod \"konnectivity-agent-9t2tv\" (UID: \"8e7057db-a92a-460c-9cbc-7f9815afa729\") " pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684520 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7-hosts-file\") pod \"node-resolver-m5kxh\" (UID: \"2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7\") " pod="openshift-dns/node-resolver-m5kxh" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684674 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7-hosts-file\") pod \"node-resolver-m5kxh\" (UID: \"2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7\") " pod="openshift-dns/node-resolver-m5kxh" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e625366f-c6d4-482e-88f8-de3023a513f6-iptables-alerter-script\") pod \"iptables-alerter-bzzg9\" (UID: \"e625366f-c6d4-482e-88f8-de3023a513f6\") " pod="openshift-network-operator/iptables-alerter-bzzg9" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684766 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-var-lib-openvswitch\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684796 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-device-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684837 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/976133c2-7307-4130-a56d-56f9727c7ba6-dbus\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684842 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e625366f-c6d4-482e-88f8-de3023a513f6-host-slash\") pod \"iptables-alerter-bzzg9\" (UID: \"e625366f-c6d4-482e-88f8-de3023a513f6\") " pod="openshift-network-operator/iptables-alerter-bzzg9" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-run\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684937 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e625366f-c6d4-482e-88f8-de3023a513f6-host-slash\") pod \"iptables-alerter-bzzg9\" (UID: \"e625366f-c6d4-482e-88f8-de3023a513f6\") " pod="openshift-network-operator/iptables-alerter-bzzg9" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684968 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-var-lib-kubelet\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.684999 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-system-cni-dir\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685027 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685057 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znqhg\" (UniqueName: \"kubernetes.io/projected/2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7-kube-api-access-znqhg\") pod \"node-resolver-m5kxh\" (UID: \"2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7\") " pod="openshift-dns/node-resolver-m5kxh" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685126 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-log-socket\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.697236 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685157 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685187 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-registration-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-var-lib-cni-bin\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685245 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-var-lib-kubelet\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685246 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/976133c2-7307-4130-a56d-56f9727c7ba6-dbus\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685451 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-tuned\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685501 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6171ceb6-38d3-46f3-a504-42094bd28cde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685525 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-etc-openvswitch\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685545 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-run-netns\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685565 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-sysconfig\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685588 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-systemd-units\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685621 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fec2b764-2853-4ff8-95ff-b0264fb8e16e-env-overrides\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685645 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-system-cni-dir\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685707 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-conf-dir\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685733 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-etc-kubernetes\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685752 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0045faa6-bed2-460c-b8f7-eaf7f30a7685-tmp\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7-tmp-dir\") pod \"node-resolver-m5kxh\" (UID: \"2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7\") " pod="openshift-dns/node-resolver-m5kxh" Jul 10 22:21:54.698384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685811 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-run-systemd\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-run-ovn\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685853 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-kubernetes\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685878 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5bgc\" (UniqueName: \"kubernetes.io/projected/1d1f9ea3-e113-4617-bd09-fc1b793a2bbe-kube-api-access-j5bgc\") pod \"node-ca-7hj6f\" (UID: \"1d1f9ea3-e113-4617-bd09-fc1b793a2bbe\") " pod="openshift-image-registry/node-ca-7hj6f" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685901 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5vzl\" (UniqueName: \"kubernetes.io/projected/81ca9b9c-e95f-4ac1-9e97-96169222847b-kube-api-access-t5vzl\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685924 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e7057db-a92a-460c-9cbc-7f9815afa729-konnectivity-ca\") pod \"konnectivity-agent-9t2tv\" (UID: \"8e7057db-a92a-460c-9cbc-7f9815afa729\") " pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685945 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-run-netns\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685969 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-hostroot\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685980 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e625366f-c6d4-482e-88f8-de3023a513f6-iptables-alerter-script\") pod \"iptables-alerter-bzzg9\" (UID: \"e625366f-c6d4-482e-88f8-de3023a513f6\") " pod="openshift-network-operator/iptables-alerter-bzzg9" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685994 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7-tmp-dir\") pod \"node-resolver-m5kxh\" (UID: \"2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7\") " pod="openshift-dns/node-resolver-m5kxh" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.685989 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-sysctl-conf\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.686098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-lib-modules\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.686151 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6171ceb6-38d3-46f3-a504-42094bd28cde-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.686189 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-os-release\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.686221 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-var-lib-cni-multus\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.686513 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8e7057db-a92a-460c-9cbc-7f9815afa729-konnectivity-ca\") pod \"konnectivity-agent-9t2tv\" (UID: \"8e7057db-a92a-460c-9cbc-7f9815afa729\") " pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.686351 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbw4m\" (UniqueName: \"kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m\") pod \"network-check-target-98cjk\" (UID: \"43f8635d-832b-42d5-932b-5e06411e985a\") " pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:21:54.699920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.687372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/976133c2-7307-4130-a56d-56f9727c7ba6-kubelet-config\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.687329 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/976133c2-7307-4130-a56d-56f9727c7ba6-kubelet-config\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.688670 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.688696 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.688721 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-cni-dir\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.688744 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-socket-dir-parent\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.688766 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-run-ovn-kubernetes\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.688788 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv44n\" (UniqueName: \"kubernetes.io/projected/fec2b764-2853-4ff8-95ff-b0264fb8e16e-kube-api-access-pv44n\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.688829 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zzmq\" (UniqueName: \"kubernetes.io/projected/7c6ad821-e27e-49a2-85ec-354c19403be5-kube-api-access-8zzmq\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.688853 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-modprobe-d\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.688876 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-sys\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.688900 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-os-release\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.689779 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8e7057db-a92a-460c-9cbc-7f9815afa729-agent-certs\") pod \"konnectivity-agent-9t2tv\" (UID: \"8e7057db-a92a-460c-9cbc-7f9815afa729\") " pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:54.696259 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:54.696284 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:54.696297 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nbw4m for pod openshift-network-diagnostics/network-check-target-98cjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:21:54.701241 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:54.696372 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m podName:43f8635d-832b-42d5-932b-5e06411e985a nodeName:}" failed. No retries permitted until 2026-07-10 22:21:55.196342323 +0000 UTC m=+3.044624192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nbw4m" (UniqueName: "kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m") pod "network-check-target-98cjk" (UID: "43f8635d-832b-42d5-932b-5e06411e985a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:21:54.703338 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.700247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znqhg\" (UniqueName: \"kubernetes.io/projected/2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7-kube-api-access-znqhg\") pod \"node-resolver-m5kxh\" (UID: \"2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7\") " pod="openshift-dns/node-resolver-m5kxh" Jul 10 22:21:54.703338 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.701808 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dchj\" (UniqueName: \"kubernetes.io/projected/e625366f-c6d4-482e-88f8-de3023a513f6-kube-api-access-6dchj\") pod \"iptables-alerter-bzzg9\" (UID: \"e625366f-c6d4-482e-88f8-de3023a513f6\") " pod="openshift-network-operator/iptables-alerter-bzzg9" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.735845 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-116.ec2.internal" event={"ID":"419dafcf2fcd506b2efc42482d8111c7","Type":"ContainerStarted","Data":"1327b4fb6102dc0589c76ca36322b4a4f2c29d226df2171bba646e90ab2f5b08"} Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.737340 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" event={"ID":"f08f323b708e7c395ca2dce63145ea16","Type":"ContainerStarted","Data":"6bfcc661287cfebe7338212ec8cf0da23b85cc67ec9fcafc642d230f79cde32e"} Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789224 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-var-lib-kubelet\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-tuned\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6171ceb6-38d3-46f3-a504-42094bd28cde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-etc-openvswitch\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789354 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-etc-openvswitch\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789361 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-var-lib-kubelet\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-run-netns\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789467 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-sysconfig\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789497 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-systemd-units\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789522 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fec2b764-2853-4ff8-95ff-b0264fb8e16e-env-overrides\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789535 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-sysconfig\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789547 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-system-cni-dir\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789580 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-conf-dir\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789585 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-systemd-units\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789605 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-etc-kubernetes\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.810381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789610 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-run-netns\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789638 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-conf-dir\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0045faa6-bed2-460c-b8f7-eaf7f30a7685-tmp\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789666 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-etc-kubernetes\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789658 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-system-cni-dir\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789686 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-run-systemd\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-run-ovn\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789734 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-kubernetes\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789736 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-run-systemd\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789760 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5bgc\" (UniqueName: \"kubernetes.io/projected/1d1f9ea3-e113-4617-bd09-fc1b793a2bbe-kube-api-access-j5bgc\") pod \"node-ca-7hj6f\" (UID: \"1d1f9ea3-e113-4617-bd09-fc1b793a2bbe\") " pod="openshift-image-registry/node-ca-7hj6f" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5vzl\" (UniqueName: \"kubernetes.io/projected/81ca9b9c-e95f-4ac1-9e97-96169222847b-kube-api-access-t5vzl\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789822 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-run-ovn\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-run-netns\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-hostroot\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-sysctl-conf\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789875 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-kubernetes\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6171ceb6-38d3-46f3-a504-42094bd28cde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789959 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-hostroot\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.811247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.789957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-lib-modules\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790000 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6171ceb6-38d3-46f3-a504-42094bd28cde-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790019 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fec2b764-2853-4ff8-95ff-b0264fb8e16e-env-overrides\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-os-release\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790028 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-run-netns\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790076 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-var-lib-cni-multus\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790087 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-os-release\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790086 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-sysctl-conf\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790109 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-lib-modules\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790122 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790164 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-var-lib-cni-multus\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790174 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-cni-dir\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790202 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-socket-dir-parent\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790215 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790240 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-run-ovn-kubernetes\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790284 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-etc-selinux\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.812032 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv44n\" (UniqueName: \"kubernetes.io/projected/fec2b764-2853-4ff8-95ff-b0264fb8e16e-kube-api-access-pv44n\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790311 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-run-ovn-kubernetes\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790327 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-cni-dir\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790329 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zzmq\" (UniqueName: \"kubernetes.io/projected/7c6ad821-e27e-49a2-85ec-354c19403be5-kube-api-access-8zzmq\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790367 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-modprobe-d\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-sys\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-os-release\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790497 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-os-release\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790500 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-socket-dir-parent\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790506 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-modprobe-d\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790547 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-sys\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790577 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fec2b764-2853-4ff8-95ff-b0264fb8e16e-ovn-node-metrics-cert\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790604 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-daemon-config\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fec2b764-2853-4ff8-95ff-b0264fb8e16e-ovnkube-script-lib\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790652 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-run-openvswitch\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790670 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-cni-netd\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790694 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-sys-fs\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790717 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c6ad821-e27e-49a2-85ec-354c19403be5-cni-binary-copy\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.813011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790724 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-run-openvswitch\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790740 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-cni-bin\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-run-k8s-cni-cncf-io\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790790 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-systemd\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790815 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-node-log\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-socket-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790869 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6171ceb6-38d3-46f3-a504-42094bd28cde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790927 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-slash\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-run-multus-certs\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.790991 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-sysctl-d\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791017 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g26jp\" (UniqueName: \"kubernetes.io/projected/0045faa6-bed2-460c-b8f7-eaf7f30a7685-kube-api-access-g26jp\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d1f9ea3-e113-4617-bd09-fc1b793a2bbe-host\") pod \"node-ca-7hj6f\" (UID: \"1d1f9ea3-e113-4617-bd09-fc1b793a2bbe\") " pod="openshift-image-registry/node-ca-7hj6f" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791062 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d1f9ea3-e113-4617-bd09-fc1b793a2bbe-serviceca\") pod \"node-ca-7hj6f\" (UID: \"1d1f9ea3-e113-4617-bd09-fc1b793a2bbe\") " pod="openshift-image-registry/node-ca-7hj6f" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791089 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-kubelet\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791122 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fec2b764-2853-4ff8-95ff-b0264fb8e16e-ovnkube-config\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791146 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6171ceb6-38d3-46f3-a504-42094bd28cde-cni-binary-copy\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791151 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpnhz\" (UniqueName: \"kubernetes.io/projected/54dafdf9-e230-4a2f-9f33-df68421e3c5a-kube-api-access-qpnhz\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.814270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-host\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791221 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fec2b764-2853-4ff8-95ff-b0264fb8e16e-ovnkube-script-lib\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791239 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-cnibin\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791249 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7c6ad821-e27e-49a2-85ec-354c19403be5-multus-daemon-config\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791266 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdp72\" (UniqueName: \"kubernetes.io/projected/6171ceb6-38d3-46f3-a504-42094bd28cde-kube-api-access-jdp72\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791281 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-cni-netd\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791294 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791311 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-host\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-cnibin\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791324 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-cnibin\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791351 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-var-lib-openvswitch\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791378 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-sys-fs\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791378 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-slash\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791377 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-device-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791433 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-device-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791450 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-run\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:54.791466 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791478 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-var-lib-kubelet\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.815250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-system-cni-dir\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791553 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:54.791606 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs podName:81ca9b9c-e95f-4ac1-9e97-96169222847b nodeName:}" failed. No retries permitted until 2026-07-10 22:21:55.291587782 +0000 UTC m=+3.139869654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs") pod "network-metrics-daemon-wbxsz" (UID: "81ca9b9c-e95f-4ac1-9e97-96169222847b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791628 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-log-socket\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791679 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-run-multus-certs\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-registration-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-var-lib-cni-bin\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791763 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6171ceb6-38d3-46f3-a504-42094bd28cde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-sysctl-d\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791797 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-var-lib-cni-bin\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791816 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-node-log\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791854 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-cnibin\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791856 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-cni-bin\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7c6ad821-e27e-49a2-85ec-354c19403be5-host-run-k8s-cni-cncf-io\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.816692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791906 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-var-lib-openvswitch\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.791945 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-systemd\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792146 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-log-socket\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792195 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792222 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-registration-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792248 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-run\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792272 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1d1f9ea3-e113-4617-bd09-fc1b793a2bbe-host\") pod \"node-ca-7hj6f\" (UID: \"1d1f9ea3-e113-4617-bd09-fc1b793a2bbe\") " pod="openshift-image-registry/node-ca-7hj6f" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792320 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0045faa6-bed2-460c-b8f7-eaf7f30a7685-var-lib-kubelet\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792355 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/54dafdf9-e230-4a2f-9f33-df68421e3c5a-socket-dir\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792346 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d1f9ea3-e113-4617-bd09-fc1b793a2bbe-serviceca\") pod \"node-ca-7hj6f\" (UID: \"1d1f9ea3-e113-4617-bd09-fc1b793a2bbe\") " pod="openshift-image-registry/node-ca-7hj6f" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792378 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fec2b764-2853-4ff8-95ff-b0264fb8e16e-host-kubelet\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6171ceb6-38d3-46f3-a504-42094bd28cde-system-cni-dir\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792464 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fec2b764-2853-4ff8-95ff-b0264fb8e16e-ovnkube-config\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792714 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c6ad821-e27e-49a2-85ec-354c19403be5-cni-binary-copy\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.792798 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0045faa6-bed2-460c-b8f7-eaf7f30a7685-tmp\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.794200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fec2b764-2853-4ff8-95ff-b0264fb8e16e-ovn-node-metrics-cert\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.799859 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0045faa6-bed2-460c-b8f7-eaf7f30a7685-etc-tuned\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.801825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5vzl\" (UniqueName: \"kubernetes.io/projected/81ca9b9c-e95f-4ac1-9e97-96169222847b-kube-api-access-t5vzl\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:54.817532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.806192 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv44n\" (UniqueName: \"kubernetes.io/projected/fec2b764-2853-4ff8-95ff-b0264fb8e16e-kube-api-access-pv44n\") pod \"ovnkube-node-tkx48\" (UID: \"fec2b764-2853-4ff8-95ff-b0264fb8e16e\") " pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.818454 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.806290 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g26jp\" (UniqueName: \"kubernetes.io/projected/0045faa6-bed2-460c-b8f7-eaf7f30a7685-kube-api-access-g26jp\") pod \"tuned-j97hm\" (UID: \"0045faa6-bed2-460c-b8f7-eaf7f30a7685\") " pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.818454 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.806430 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zzmq\" (UniqueName: \"kubernetes.io/projected/7c6ad821-e27e-49a2-85ec-354c19403be5-kube-api-access-8zzmq\") pod \"multus-fmj2g\" (UID: \"7c6ad821-e27e-49a2-85ec-354c19403be5\") " pod="openshift-multus/multus-fmj2g" Jul 10 22:21:54.818454 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.806791 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5bgc\" (UniqueName: \"kubernetes.io/projected/1d1f9ea3-e113-4617-bd09-fc1b793a2bbe-kube-api-access-j5bgc\") pod \"node-ca-7hj6f\" (UID: \"1d1f9ea3-e113-4617-bd09-fc1b793a2bbe\") " pod="openshift-image-registry/node-ca-7hj6f" Jul 10 22:21:54.818454 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.807244 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdp72\" (UniqueName: \"kubernetes.io/projected/6171ceb6-38d3-46f3-a504-42094bd28cde-kube-api-access-jdp72\") pod \"multus-additional-cni-plugins-nfvl2\" (UID: \"6171ceb6-38d3-46f3-a504-42094bd28cde\") " pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.818454 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.807634 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpnhz\" (UniqueName: \"kubernetes.io/projected/54dafdf9-e230-4a2f-9f33-df68421e3c5a-kube-api-access-qpnhz\") pod \"aws-ebs-csi-driver-node-6xtmz\" (UID: \"54dafdf9-e230-4a2f-9f33-df68421e3c5a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.950214 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.875704 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:21:54.950214 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.888683 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-m5kxh" Jul 10 22:21:54.950214 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.908124 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bzzg9" Jul 10 22:21:54.950214 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.913014 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:21:54.950214 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.928247 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" Jul 10 22:21:54.950214 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.936368 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-j97hm" Jul 10 22:21:54.950214 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.946570 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7hj6f" Jul 10 22:21:54.959011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.956214 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" Jul 10 22:21:54.962012 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:54.960608 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fmj2g" Jul 10 22:21:55.296111 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.195361 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:55.296111 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:55.195517 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:21:55.296111 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:55.195580 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret podName:976133c2-7307-4130-a56d-56f9727c7ba6 nodeName:}" failed. No retries permitted until 2026-07-10 22:21:56.195562528 +0000 UTC m=+4.043844385 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret") pod "global-pull-secret-syncer-krhr5" (UID: "976133c2-7307-4130-a56d-56f9727c7ba6") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:21:55.297992 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.296569 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:55.297992 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.296607 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbw4m\" (UniqueName: \"kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m\") pod \"network-check-target-98cjk\" (UID: \"43f8635d-832b-42d5-932b-5e06411e985a\") " pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:21:55.297992 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:55.296722 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:21:55.297992 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:55.296738 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:21:55.297992 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:55.296739 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:21:55.297992 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:55.296816 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs podName:81ca9b9c-e95f-4ac1-9e97-96169222847b nodeName:}" failed. No retries permitted until 2026-07-10 22:21:56.296796089 +0000 UTC m=+4.145077943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs") pod "network-metrics-daemon-wbxsz" (UID: "81ca9b9c-e95f-4ac1-9e97-96169222847b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:21:55.297992 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:55.296747 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nbw4m for pod openshift-network-diagnostics/network-check-target-98cjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:21:55.297992 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:55.296890 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m podName:43f8635d-832b-42d5-932b-5e06411e985a nodeName:}" failed. No retries permitted until 2026-07-10 22:21:56.296874609 +0000 UTC m=+4.145156465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbw4m" (UniqueName: "kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m") pod "network-check-target-98cjk" (UID: "43f8635d-832b-42d5-932b-5e06411e985a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:21:55.510945 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:55.510908 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb9e8fd_ed42_42c3_8174_34e2ac5a51d7.slice/crio-d04eabdee9b3b9a156b574820e84544ead291a9526fc7452a0cb57a48990b025 WatchSource:0}: Error finding container d04eabdee9b3b9a156b574820e84544ead291a9526fc7452a0cb57a48990b025: Status 404 returned error can't find the container with id d04eabdee9b3b9a156b574820e84544ead291a9526fc7452a0cb57a48990b025 Jul 10 22:21:55.514009 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:55.512246 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7057db_a92a_460c_9cbc_7f9815afa729.slice/crio-3ab6a354494f3a2d4f5319ad42a42b3a4d191d7e37b61c617ae5e81c2a5e1745 WatchSource:0}: Error finding container 3ab6a354494f3a2d4f5319ad42a42b3a4d191d7e37b61c617ae5e81c2a5e1745: Status 404 returned error can't find the container with id 3ab6a354494f3a2d4f5319ad42a42b3a4d191d7e37b61c617ae5e81c2a5e1745 Jul 10 22:21:55.514009 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:55.513365 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode625366f_c6d4_482e_88f8_de3023a513f6.slice/crio-358906ae71f6d809b6d1be871727c17c166e3d10760ee3c0451678865bf90906 WatchSource:0}: Error finding container 358906ae71f6d809b6d1be871727c17c166e3d10760ee3c0451678865bf90906: Status 404 returned error can't find the container with id 358906ae71f6d809b6d1be871727c17c166e3d10760ee3c0451678865bf90906 Jul 10 22:21:55.515830 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:55.515805 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1f9ea3_e113_4617_bd09_fc1b793a2bbe.slice/crio-532b5e8bf148ec111cc08e0df38fcb3d88476f5bc921e8da0b3f6f36dffee624 WatchSource:0}: Error finding container 532b5e8bf148ec111cc08e0df38fcb3d88476f5bc921e8da0b3f6f36dffee624: Status 404 returned error can't find the container with id 532b5e8bf148ec111cc08e0df38fcb3d88476f5bc921e8da0b3f6f36dffee624 Jul 10 22:21:55.519841 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:55.519822 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dafdf9_e230_4a2f_9f33_df68421e3c5a.slice/crio-7a356107d77fb22dfffb595cedbe6ceacf0457163746bfddbdaf9ccb89d78b38 WatchSource:0}: Error finding container 7a356107d77fb22dfffb595cedbe6ceacf0457163746bfddbdaf9ccb89d78b38: Status 404 returned error can't find the container with id 7a356107d77fb22dfffb595cedbe6ceacf0457163746bfddbdaf9ccb89d78b38 Jul 10 22:21:55.540626 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:55.540607 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec2b764_2853_4ff8_95ff_b0264fb8e16e.slice/crio-07ee42fd914645fb8408b249754b95a21493028be9cbc84073a7a36e876cc881 WatchSource:0}: Error finding container 07ee42fd914645fb8408b249754b95a21493028be9cbc84073a7a36e876cc881: Status 404 returned error can't find the container with id 07ee42fd914645fb8408b249754b95a21493028be9cbc84073a7a36e876cc881 Jul 10 22:21:55.541747 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:55.541724 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6ad821_e27e_49a2_85ec_354c19403be5.slice/crio-97577afbb8e67e93bee80b0ce1eedd0963e449734dd48bfefcfe0f963acd4b5c WatchSource:0}: Error finding container 97577afbb8e67e93bee80b0ce1eedd0963e449734dd48bfefcfe0f963acd4b5c: Status 404 returned error can't find the container with id 97577afbb8e67e93bee80b0ce1eedd0963e449734dd48bfefcfe0f963acd4b5c Jul 10 22:21:55.542397 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:55.542377 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6171ceb6_38d3_46f3_a504_42094bd28cde.slice/crio-fb51b3f89f54d523d836a644681cafe62f30169c5bc3960a063774c7195fc7d7 WatchSource:0}: Error finding container fb51b3f89f54d523d836a644681cafe62f30169c5bc3960a063774c7195fc7d7: Status 404 returned error can't find the container with id fb51b3f89f54d523d836a644681cafe62f30169c5bc3960a063774c7195fc7d7 Jul 10 22:21:55.543853 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:21:55.543832 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0045faa6_bed2_460c_b8f7_eaf7f30a7685.slice/crio-48b145fe9bd09809ca143fdd6de94c783ce61f85f61ef066848982e2e42e34dc WatchSource:0}: Error finding container 48b145fe9bd09809ca143fdd6de94c783ce61f85f61ef066848982e2e42e34dc: Status 404 returned error can't find the container with id 48b145fe9bd09809ca143fdd6de94c783ce61f85f61ef066848982e2e42e34dc Jul 10 22:21:55.626394 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.626346 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 22:16:54 +0000 UTC" deadline="2028-04-14 17:50:56.165042788 +0000 UTC" Jul 10 22:21:55.626394 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.626396 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15451h29m0.538651453s" Jul 10 22:21:55.633686 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.633651 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 22:21:55.719962 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.719931 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:55.719962 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.719968 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:55.720179 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:55.720060 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:21:55.720227 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:55.720176 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:21:55.740315 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.740292 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" event={"ID":"fec2b764-2853-4ff8-95ff-b0264fb8e16e","Type":"ContainerStarted","Data":"07ee42fd914645fb8408b249754b95a21493028be9cbc84073a7a36e876cc881"} Jul 10 22:21:55.741686 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.741660 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j97hm" event={"ID":"0045faa6-bed2-460c-b8f7-eaf7f30a7685","Type":"ContainerStarted","Data":"48b145fe9bd09809ca143fdd6de94c783ce61f85f61ef066848982e2e42e34dc"} Jul 10 22:21:55.743046 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.743019 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" event={"ID":"6171ceb6-38d3-46f3-a504-42094bd28cde","Type":"ContainerStarted","Data":"fb51b3f89f54d523d836a644681cafe62f30169c5bc3960a063774c7195fc7d7"} Jul 10 22:21:55.744325 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.744301 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fmj2g" event={"ID":"7c6ad821-e27e-49a2-85ec-354c19403be5","Type":"ContainerStarted","Data":"97577afbb8e67e93bee80b0ce1eedd0963e449734dd48bfefcfe0f963acd4b5c"} Jul 10 22:21:55.745499 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.745474 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" event={"ID":"54dafdf9-e230-4a2f-9f33-df68421e3c5a","Type":"ContainerStarted","Data":"7a356107d77fb22dfffb595cedbe6ceacf0457163746bfddbdaf9ccb89d78b38"} Jul 10 22:21:55.746644 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.746612 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7hj6f" event={"ID":"1d1f9ea3-e113-4617-bd09-fc1b793a2bbe","Type":"ContainerStarted","Data":"532b5e8bf148ec111cc08e0df38fcb3d88476f5bc921e8da0b3f6f36dffee624"} Jul 10 22:21:55.747934 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.747899 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bzzg9" event={"ID":"e625366f-c6d4-482e-88f8-de3023a513f6","Type":"ContainerStarted","Data":"358906ae71f6d809b6d1be871727c17c166e3d10760ee3c0451678865bf90906"} Jul 10 22:21:55.749047 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.749024 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9t2tv" event={"ID":"8e7057db-a92a-460c-9cbc-7f9815afa729","Type":"ContainerStarted","Data":"3ab6a354494f3a2d4f5319ad42a42b3a4d191d7e37b61c617ae5e81c2a5e1745"} Jul 10 22:21:55.751143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:55.750951 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m5kxh" event={"ID":"2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7","Type":"ContainerStarted","Data":"d04eabdee9b3b9a156b574820e84544ead291a9526fc7452a0cb57a48990b025"} Jul 10 22:21:56.205086 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:56.204382 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:56.205086 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:56.204632 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:21:56.205086 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:56.204717 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret podName:976133c2-7307-4130-a56d-56f9727c7ba6 nodeName:}" failed. No retries permitted until 2026-07-10 22:21:58.20469736 +0000 UTC m=+6.052979222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret") pod "global-pull-secret-syncer-krhr5" (UID: "976133c2-7307-4130-a56d-56f9727c7ba6") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:21:56.305432 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:56.305393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:56.305620 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:56.305507 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbw4m\" (UniqueName: \"kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m\") pod \"network-check-target-98cjk\" (UID: \"43f8635d-832b-42d5-932b-5e06411e985a\") " pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:21:56.305742 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:56.305694 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:21:56.305742 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:56.305716 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:21:56.305742 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:56.305729 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nbw4m for pod openshift-network-diagnostics/network-check-target-98cjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:21:56.305888 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:56.305784 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m podName:43f8635d-832b-42d5-932b-5e06411e985a nodeName:}" failed. No retries permitted until 2026-07-10 22:21:58.305766675 +0000 UTC m=+6.154048542 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbw4m" (UniqueName: "kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m") pod "network-check-target-98cjk" (UID: "43f8635d-832b-42d5-932b-5e06411e985a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:21:56.306210 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:56.306189 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:21:56.306270 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:56.306249 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs podName:81ca9b9c-e95f-4ac1-9e97-96169222847b nodeName:}" failed. No retries permitted until 2026-07-10 22:21:58.306231862 +0000 UTC m=+6.154513737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs") pod "network-metrics-daemon-wbxsz" (UID: "81ca9b9c-e95f-4ac1-9e97-96169222847b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:21:56.626593 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:56.626519 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 22:16:54 +0000 UTC" deadline="2028-03-29 12:12:39.140884506 +0000 UTC" Jul 10 22:21:56.626593 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:56.626569 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15061h50m42.514318384s" Jul 10 22:21:56.720569 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:56.720545 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:21:56.720695 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:56.720665 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:21:56.765359 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:56.765324 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-116.ec2.internal" event={"ID":"419dafcf2fcd506b2efc42482d8111c7","Type":"ContainerStarted","Data":"c999a8fc3461d2acc129c5a5db3693c681edb42736265798185af4c53edf07c1"} Jul 10 22:21:56.771421 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:56.771368 2559 generic.go:356] "Generic (PLEG): container finished" podID="f08f323b708e7c395ca2dce63145ea16" containerID="f9abec3ef1b65ec91de65be11bb6479a3ca0e33b3583e433d544290a817f7244" exitCode=0 Jul 10 22:21:56.771547 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:56.771457 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" event={"ID":"f08f323b708e7c395ca2dce63145ea16","Type":"ContainerDied","Data":"f9abec3ef1b65ec91de65be11bb6479a3ca0e33b3583e433d544290a817f7244"} Jul 10 22:21:56.780519 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:56.780473 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-116.ec2.internal" podStartSLOduration=2.780455132 podStartE2EDuration="2.780455132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:21:56.780055552 +0000 UTC m=+4.628337618" watchObservedRunningTime="2026-07-10 22:21:56.780455132 +0000 UTC m=+4.628737002" Jul 10 22:21:57.721206 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:57.720263 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:57.721206 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:57.720398 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:21:57.721206 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:57.720801 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:57.721206 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:57.720897 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:21:57.796798 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:57.796722 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" event={"ID":"f08f323b708e7c395ca2dce63145ea16","Type":"ContainerStarted","Data":"eae80e8e73aff071d78df4affe03714c85fd6d7269902d7d751dc1dc43d6dfd7"} Jul 10 22:21:58.226609 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:58.225999 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:58.226609 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:58.226159 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:21:58.226609 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:58.226245 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret podName:976133c2-7307-4130-a56d-56f9727c7ba6 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:02.22622667 +0000 UTC m=+10.074508526 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret") pod "global-pull-secret-syncer-krhr5" (UID: "976133c2-7307-4130-a56d-56f9727c7ba6") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:21:58.327539 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:58.326935 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbw4m\" (UniqueName: \"kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m\") pod \"network-check-target-98cjk\" (UID: \"43f8635d-832b-42d5-932b-5e06411e985a\") " pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:21:58.327539 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:58.327009 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:58.327539 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:58.327099 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:21:58.327539 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:58.327160 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs podName:81ca9b9c-e95f-4ac1-9e97-96169222847b nodeName:}" failed. No retries permitted until 2026-07-10 22:22:02.327140154 +0000 UTC m=+10.175422029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs") pod "network-metrics-daemon-wbxsz" (UID: "81ca9b9c-e95f-4ac1-9e97-96169222847b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:21:58.327539 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:58.327102 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:21:58.327539 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:58.327187 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:21:58.327539 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:58.327200 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nbw4m for pod openshift-network-diagnostics/network-check-target-98cjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:21:58.327539 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:58.327238 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m podName:43f8635d-832b-42d5-932b-5e06411e985a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:02.327224847 +0000 UTC m=+10.175506718 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbw4m" (UniqueName: "kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m") pod "network-check-target-98cjk" (UID: "43f8635d-832b-42d5-932b-5e06411e985a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:21:58.722734 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:58.722190 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:21:58.722734 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:58.722333 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:21:59.719575 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:59.719546 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:21:59.719756 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:59.719667 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:21:59.719967 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:21:59.719948 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:21:59.720102 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:21:59.720082 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:00.719796 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:00.719759 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:00.720292 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:00.719896 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:01.720195 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:01.720155 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:01.720624 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:01.720295 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:01.720665 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:01.720645 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:01.720729 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:01.720706 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:02.268331 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:02.268292 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:02.268532 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:02.268473 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:02.268532 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:02.268529 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret podName:976133c2-7307-4130-a56d-56f9727c7ba6 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:10.268510517 +0000 UTC m=+18.116792384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret") pod "global-pull-secret-syncer-krhr5" (UID: "976133c2-7307-4130-a56d-56f9727c7ba6") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:02.369010 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:02.368968 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbw4m\" (UniqueName: \"kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m\") pod \"network-check-target-98cjk\" (UID: \"43f8635d-832b-42d5-932b-5e06411e985a\") " pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:02.369200 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:02.369051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:02.369200 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:02.369183 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:02.369303 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:02.369240 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs podName:81ca9b9c-e95f-4ac1-9e97-96169222847b nodeName:}" failed. No retries permitted until 2026-07-10 22:22:10.369220985 +0000 UTC m=+18.217502852 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs") pod "network-metrics-daemon-wbxsz" (UID: "81ca9b9c-e95f-4ac1-9e97-96169222847b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:02.369651 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:02.369523 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:22:02.369651 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:02.369554 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:22:02.369651 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:02.369568 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nbw4m for pod openshift-network-diagnostics/network-check-target-98cjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:02.369651 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:02.369628 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m podName:43f8635d-832b-42d5-932b-5e06411e985a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:10.369610372 +0000 UTC m=+18.217892241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbw4m" (UniqueName: "kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m") pod "network-check-target-98cjk" (UID: "43f8635d-832b-42d5-932b-5e06411e985a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:02.720634 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:02.720602 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:02.721101 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:02.720708 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:03.719875 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:03.719767 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:03.719875 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:03.719802 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:03.720060 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:03.719874 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:03.720060 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:03.719999 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:04.720181 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:04.720148 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:04.720657 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:04.720271 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:05.719368 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:05.719333 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:05.719368 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:05.719359 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:05.719614 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:05.719460 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:05.719614 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:05.719571 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:06.719964 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:06.719924 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:06.720385 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:06.720061 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:07.719866 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:07.719829 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:07.719866 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:07.719864 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:07.720373 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:07.719939 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:07.720373 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:07.720066 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:08.720161 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:08.720125 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:08.720611 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:08.720238 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:09.719872 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:09.719827 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:09.719872 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:09.719867 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:09.720162 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:09.719982 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:09.720162 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:09.720107 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:10.330169 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:10.330128 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:10.330352 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:10.330300 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:10.330396 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:10.330365 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret podName:976133c2-7307-4130-a56d-56f9727c7ba6 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:26.330350136 +0000 UTC m=+34.178632008 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret") pod "global-pull-secret-syncer-krhr5" (UID: "976133c2-7307-4130-a56d-56f9727c7ba6") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:10.431537 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:10.431500 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:10.431707 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:10.431573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbw4m\" (UniqueName: \"kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m\") pod \"network-check-target-98cjk\" (UID: \"43f8635d-832b-42d5-932b-5e06411e985a\") " pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:10.431707 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:10.431668 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:10.431828 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:10.431716 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:22:10.431828 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:10.431731 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:22:10.431828 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:10.431743 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nbw4m for pod openshift-network-diagnostics/network-check-target-98cjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:10.431828 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:10.431750 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs podName:81ca9b9c-e95f-4ac1-9e97-96169222847b nodeName:}" failed. No retries permitted until 2026-07-10 22:22:26.431728348 +0000 UTC m=+34.280010203 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs") pod "network-metrics-daemon-wbxsz" (UID: "81ca9b9c-e95f-4ac1-9e97-96169222847b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:10.431828 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:10.431789 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m podName:43f8635d-832b-42d5-932b-5e06411e985a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:26.431774237 +0000 UTC m=+34.280056094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbw4m" (UniqueName: "kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m") pod "network-check-target-98cjk" (UID: "43f8635d-832b-42d5-932b-5e06411e985a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:10.720406 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:10.720366 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:10.720853 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:10.720520 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:11.719918 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:11.719885 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:11.720076 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:11.719901 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:11.720076 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:11.720026 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:11.720189 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:11.720074 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:12.722344 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:12.722318 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:12.722810 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:12.722408 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:13.720134 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:13.720099 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:13.720313 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:13.720152 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:13.720313 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:13.720242 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:13.720426 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:13.720379 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:14.720463 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.720357 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:14.721080 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:14.720504 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:14.829814 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.829780 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9t2tv" event={"ID":"8e7057db-a92a-460c-9cbc-7f9815afa729","Type":"ContainerStarted","Data":"2b3dad0782acdd62a94d9274de4894ca76a5e72c10fed9c9dff7c32343cf7c19"} Jul 10 22:22:14.831500 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.831469 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-m5kxh" event={"ID":"2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7","Type":"ContainerStarted","Data":"5768d8584ba4c3c2c914f6968da57ead92d55db51b38e0cb29667a2c1a116a59"} Jul 10 22:22:14.832868 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.832845 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" event={"ID":"fec2b764-2853-4ff8-95ff-b0264fb8e16e","Type":"ContainerStarted","Data":"4f18003b35833ce94d94993bb833b4da678a5f910310f221365691953dfd6022"} Jul 10 22:22:14.834257 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.834230 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-j97hm" event={"ID":"0045faa6-bed2-460c-b8f7-eaf7f30a7685","Type":"ContainerStarted","Data":"18e613d7e95a52b95501ed4f00de34f1d3c19d10aaf061c8ca7243d77a75724f"} Jul 10 22:22:14.835771 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.835743 2559 generic.go:356] "Generic (PLEG): container finished" podID="6171ceb6-38d3-46f3-a504-42094bd28cde" containerID="ec225fefd7fa3edd3828bd46d2d739d3e17a47a1f807af9f89dabe520ed615ad" exitCode=0 Jul 10 22:22:14.835867 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.835824 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" event={"ID":"6171ceb6-38d3-46f3-a504-42094bd28cde","Type":"ContainerDied","Data":"ec225fefd7fa3edd3828bd46d2d739d3e17a47a1f807af9f89dabe520ed615ad"} Jul 10 22:22:14.837355 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.837329 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fmj2g" event={"ID":"7c6ad821-e27e-49a2-85ec-354c19403be5","Type":"ContainerStarted","Data":"cc7565d90f5b71f874731c3470a404b131f49626dc6fb84d3993315ec17b8965"} Jul 10 22:22:14.838938 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.838915 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" event={"ID":"54dafdf9-e230-4a2f-9f33-df68421e3c5a","Type":"ContainerStarted","Data":"f5a5bbf04d26b7620a087c91242ad8d12fe793ab8b576a139a2bbae64a6f5622"} Jul 10 22:22:14.840333 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.840307 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7hj6f" event={"ID":"1d1f9ea3-e113-4617-bd09-fc1b793a2bbe","Type":"ContainerStarted","Data":"6af06a03dbf424f2cb717c3eddfb6e1d9b7ca6e6ff9e28a049469ccbbdf421a0"} Jul 10 22:22:14.858022 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.857980 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-116.ec2.internal" podStartSLOduration=20.857966503 podStartE2EDuration="20.857966503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:21:57.81161532 +0000 UTC m=+5.659897195" watchObservedRunningTime="2026-07-10 22:22:14.857966503 +0000 UTC m=+22.706248379" Jul 10 22:22:14.871488 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.871421 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-m5kxh" podStartSLOduration=2.936994462 podStartE2EDuration="21.871403132s" totalImagesPullingTime="18.93440867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:14.871165249 +0000 UTC m=+22.719447126" watchObservedRunningTime="2026-07-10 22:22:14.871403132 +0000 UTC m=+22.719685006" Jul 10 22:22:14.887246 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.887211 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-j97hm" podStartSLOduration=2.969185295 podStartE2EDuration="21.887197596s" totalImagesPullingTime="18.918012301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:14.886904683 +0000 UTC m=+22.735186561" watchObservedRunningTime="2026-07-10 22:22:14.887197596 +0000 UTC m=+22.735479502" Jul 10 22:22:14.902575 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.902542 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7hj6f" podStartSLOduration=7.587939454 podStartE2EDuration="21.902528414s" totalImagesPullingTime="14.31458896s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:14.902280966 +0000 UTC m=+22.750562844" watchObservedRunningTime="2026-07-10 22:22:14.902528414 +0000 UTC m=+22.750810298" Jul 10 22:22:14.938473 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.938416 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fmj2g" podStartSLOduration=3.006657785 podStartE2EDuration="21.938401729s" totalImagesPullingTime="18.931743944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:14.938043662 +0000 UTC m=+22.786325538" watchObservedRunningTime="2026-07-10 22:22:14.938401729 +0000 UTC m=+22.786683604" Jul 10 22:22:14.955261 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:14.955226 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9t2tv" podStartSLOduration=4.007584698 podStartE2EDuration="22.955212891s" totalImagesPullingTime="18.947628193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:21:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:14.955071198 +0000 UTC m=+22.803353073" watchObservedRunningTime="2026-07-10 22:22:14.955212891 +0000 UTC m=+22.803494767" Jul 10 22:22:15.719387 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:15.719360 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:15.719537 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:15.719360 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:15.719537 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:15.719495 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:15.719669 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:15.719563 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:15.836196 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:15.836170 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 22:22:15.846379 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:15.846352 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:22:15.846803 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:15.846758 2559 generic.go:356] "Generic (PLEG): container finished" podID="fec2b764-2853-4ff8-95ff-b0264fb8e16e" containerID="03a2f33918bc8a10d94369d4f0ad74ca6abd7f09a8ce36dc326a5936ce121374" exitCode=1 Jul 10 22:22:15.846988 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:15.846852 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" event={"ID":"fec2b764-2853-4ff8-95ff-b0264fb8e16e","Type":"ContainerStarted","Data":"59f59908263f87d857e6f4af607fa4673c7193aa3eeefe2beaf8f09ecb96ad1c"} Jul 10 22:22:15.846988 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:15.846926 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" event={"ID":"fec2b764-2853-4ff8-95ff-b0264fb8e16e","Type":"ContainerStarted","Data":"77bf4f519edc17ea95afcd82e1bf88766ea71f2caf2e13f65c41c4e5d65ffd94"} Jul 10 22:22:15.847135 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:15.847007 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" event={"ID":"fec2b764-2853-4ff8-95ff-b0264fb8e16e","Type":"ContainerStarted","Data":"db3b5b16d13402d92cf97a090aa881f4bdc1ff6dcb1ff3e74647bd2a52b44343"} Jul 10 22:22:15.847135 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:15.847056 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" event={"ID":"fec2b764-2853-4ff8-95ff-b0264fb8e16e","Type":"ContainerStarted","Data":"6475d084a5f0e7d706749152ca431779deaadd8a4647f3f2d9a7f3f02cb26a26"} Jul 10 22:22:15.847135 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:15.847103 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" event={"ID":"fec2b764-2853-4ff8-95ff-b0264fb8e16e","Type":"ContainerDied","Data":"03a2f33918bc8a10d94369d4f0ad74ca6abd7f09a8ce36dc326a5936ce121374"} Jul 10 22:22:15.849024 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:15.848989 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" event={"ID":"54dafdf9-e230-4a2f-9f33-df68421e3c5a","Type":"ContainerStarted","Data":"620ca8d8f68bf839b41c57cba29a4747fc306b87ced36a80a81a185f72200416"} Jul 10 22:22:16.181189 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:16.181162 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:22:16.181683 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:16.181666 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:22:16.675597 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:16.675484 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T22:22:15.836189406Z","UUID":"0fe04262-65ae-4848-8d41-fdf6771281fd","Handler":null,"Name":"","Endpoint":""} Jul 10 22:22:16.677044 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:16.677024 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 22:22:16.677175 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:16.677064 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 22:22:16.723678 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:16.723656 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:16.723849 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:16.723750 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:16.852656 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:16.852625 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bzzg9" event={"ID":"e625366f-c6d4-482e-88f8-de3023a513f6","Type":"ContainerStarted","Data":"6f903d91e0deecb47cd7254374925afea94c15cd3c5e8b2f4b86fa2c3f3ae913"} Jul 10 22:22:16.853049 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:16.852843 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:22:16.853349 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:16.853329 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9t2tv" Jul 10 22:22:16.868716 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:16.868666 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bzzg9" podStartSLOduration=4.921302845 podStartE2EDuration="23.868650377s" totalImagesPullingTime="18.947347532s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:16.868379463 +0000 UTC m=+24.716661339" watchObservedRunningTime="2026-07-10 22:22:16.868650377 +0000 UTC m=+24.716932250" Jul 10 22:22:17.719484 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:17.719455 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:17.719704 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:17.719456 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:17.719704 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:17.719561 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:17.719704 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:17.719660 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:17.860788 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:17.860761 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:22:17.861203 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:17.861171 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" event={"ID":"fec2b764-2853-4ff8-95ff-b0264fb8e16e","Type":"ContainerStarted","Data":"724e1440dc4b83ca67dce585b2dea6202c0b7c19b283732b9752b35606ede2d9"} Jul 10 22:22:17.863690 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:17.863663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" event={"ID":"54dafdf9-e230-4a2f-9f33-df68421e3c5a","Type":"ContainerStarted","Data":"187a322be48636bcd7e486353f88d01e5f0e987af5fcbc4204d7ff8927877efb"} Jul 10 22:22:17.880571 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:17.880511 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6xtmz" podStartSLOduration=3.978534487 podStartE2EDuration="24.880496779s" totalImagesPullingTime="20.901962292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:21:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:17.87965321 +0000 UTC m=+25.727935087" watchObservedRunningTime="2026-07-10 22:22:17.880496779 +0000 UTC m=+25.728778654" Jul 10 22:22:18.720179 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:18.720053 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:18.720179 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:18.720166 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:19.720402 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:19.720241 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:19.720911 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:19.720274 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:19.720911 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:19.720540 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:19.720911 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:19.720670 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:19.871171 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:19.871151 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:22:19.871721 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:19.871692 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" event={"ID":"fec2b764-2853-4ff8-95ff-b0264fb8e16e","Type":"ContainerStarted","Data":"e4a59ad4c723d5aede106ef59fd24fefb2f18c6552524ae63ac765aa35d095b3"} Jul 10 22:22:19.871983 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:19.871959 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:22:19.872058 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:19.872039 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:22:19.872127 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:19.872108 2559 scope.go:117] "RemoveContainer" containerID="03a2f33918bc8a10d94369d4f0ad74ca6abd7f09a8ce36dc326a5936ce121374" Jul 10 22:22:19.873605 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:19.873578 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" event={"ID":"6171ceb6-38d3-46f3-a504-42094bd28cde","Type":"ContainerStarted","Data":"ea93f4c7fd70f275f6c656aea0d1859d742215f316e75f9c3b17ffec4464cbba"} Jul 10 22:22:19.887321 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:19.887303 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:22:20.719875 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:20.719844 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:20.720056 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:20.719960 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:20.881791 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:20.881760 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:22:20.882257 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:20.882223 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" event={"ID":"fec2b764-2853-4ff8-95ff-b0264fb8e16e","Type":"ContainerStarted","Data":"1dc1a5fe1952ce967502adecc3b89628c0821b98fada61669844bba040295bff"} Jul 10 22:22:20.882761 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:20.882730 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:22:20.885677 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:20.885645 2559 generic.go:356] "Generic (PLEG): container finished" podID="6171ceb6-38d3-46f3-a504-42094bd28cde" containerID="ea93f4c7fd70f275f6c656aea0d1859d742215f316e75f9c3b17ffec4464cbba" exitCode=0 Jul 10 22:22:20.885781 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:20.885690 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" event={"ID":"6171ceb6-38d3-46f3-a504-42094bd28cde","Type":"ContainerDied","Data":"ea93f4c7fd70f275f6c656aea0d1859d742215f316e75f9c3b17ffec4464cbba"} Jul 10 22:22:20.907220 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:20.907197 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:22:20.925168 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:20.925126 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" podStartSLOduration=8.808986385 podStartE2EDuration="27.925114289s" totalImagesPullingTime="19.116127904s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:20.923802282 +0000 UTC m=+28.772084155" watchObservedRunningTime="2026-07-10 22:22:20.925114289 +0000 UTC m=+28.773396184" Jul 10 22:22:21.555712 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:21.555681 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-krhr5"] Jul 10 22:22:21.555968 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:21.555816 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:21.555968 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:21.555923 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:21.560026 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:21.559999 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wbxsz"] Jul 10 22:22:21.560136 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:21.560111 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:21.560238 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:21.560217 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:21.560648 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:21.560618 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-98cjk"] Jul 10 22:22:21.560719 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:21.560704 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:21.560790 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:21.560773 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:22.720678 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:22.720524 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:22.721022 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:22.720752 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:22.891529 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:22.891495 2559 generic.go:356] "Generic (PLEG): container finished" podID="6171ceb6-38d3-46f3-a504-42094bd28cde" containerID="476c3e93ae97a1700aacf740c34f8e1685fa935445bbfb879c14455a0332f250" exitCode=0 Jul 10 22:22:22.891699 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:22.891581 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" event={"ID":"6171ceb6-38d3-46f3-a504-42094bd28cde","Type":"ContainerDied","Data":"476c3e93ae97a1700aacf740c34f8e1685fa935445bbfb879c14455a0332f250"} Jul 10 22:22:23.719713 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:23.719646 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:23.719846 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:23.719738 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:23.719846 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:23.719792 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:23.719927 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:23.719866 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:24.720325 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:24.720295 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:24.721004 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:24.720402 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:24.898065 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:24.898025 2559 generic.go:356] "Generic (PLEG): container finished" podID="6171ceb6-38d3-46f3-a504-42094bd28cde" containerID="1695c13124489f2e1e1759885b847997cdcd19516fbe83fd73f814aa59cdd238" exitCode=0 Jul 10 22:22:24.898215 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:24.898077 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" event={"ID":"6171ceb6-38d3-46f3-a504-42094bd28cde","Type":"ContainerDied","Data":"1695c13124489f2e1e1759885b847997cdcd19516fbe83fd73f814aa59cdd238"} Jul 10 22:22:25.719903 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:25.719865 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:25.720072 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:25.719991 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:25.720072 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:25.720055 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:25.720207 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:25.720182 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:26.360154 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:26.360125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:26.360562 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:26.360240 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:26.360562 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:26.360307 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret podName:976133c2-7307-4130-a56d-56f9727c7ba6 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:58.360287609 +0000 UTC m=+66.208569463 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret") pod "global-pull-secret-syncer-krhr5" (UID: "976133c2-7307-4130-a56d-56f9727c7ba6") : object "kube-system"/"original-pull-secret" not registered Jul 10 22:22:26.460598 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:26.460563 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbw4m\" (UniqueName: \"kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m\") pod \"network-check-target-98cjk\" (UID: \"43f8635d-832b-42d5-932b-5e06411e985a\") " pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:26.460791 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:26.460633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:26.460791 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:26.460739 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 22:22:26.460791 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:26.460768 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 22:22:26.460791 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:26.460779 2559 projected.go:196] Error preparing data for projected volume kube-api-access-nbw4m for pod openshift-network-diagnostics/network-check-target-98cjk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:26.461010 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:26.460830 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m podName:43f8635d-832b-42d5-932b-5e06411e985a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:58.46081191 +0000 UTC m=+66.309093763 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nbw4m" (UniqueName: "kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m") pod "network-check-target-98cjk" (UID: "43f8635d-832b-42d5-932b-5e06411e985a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 22:22:26.461010 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:26.460742 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:26.461010 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:26.460895 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs podName:81ca9b9c-e95f-4ac1-9e97-96169222847b nodeName:}" failed. No retries permitted until 2026-07-10 22:22:58.460881862 +0000 UTC m=+66.309163730 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs") pod "network-metrics-daemon-wbxsz" (UID: "81ca9b9c-e95f-4ac1-9e97-96169222847b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 22:22:26.720363 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:26.720331 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:26.720566 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:26.720476 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wbxsz" podUID="81ca9b9c-e95f-4ac1-9e97-96169222847b" Jul 10 22:22:27.720356 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:27.720323 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:27.720886 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:27.720324 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:27.720886 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:27.720457 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-98cjk" podUID="43f8635d-832b-42d5-932b-5e06411e985a" Jul 10 22:22:27.720886 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:27.720559 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-krhr5" podUID="976133c2-7307-4130-a56d-56f9727c7ba6" Jul 10 22:22:28.471671 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.471643 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-116.ec2.internal" event="NodeReady" Jul 10 22:22:28.471835 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.471802 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 22:22:28.506065 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.506028 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7db6cd9c68-dft8m"] Jul 10 22:22:28.511260 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.511239 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.513826 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.513799 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 10 22:22:28.514411 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.514393 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 10 22:22:28.514651 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.514631 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 10 22:22:28.514905 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.514874 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5zdmx\"" Jul 10 22:22:28.520913 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.520808 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 10 22:22:28.525354 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.525333 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8s7sv"] Jul 10 22:22:28.528910 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.528892 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7db6cd9c68-dft8m"] Jul 10 22:22:28.529012 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.528923 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-s7t6h"] Jul 10 22:22:28.529067 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.529053 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.531272 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.531249 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t9ksf\"" Jul 10 22:22:28.531391 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.531288 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 22:22:28.531391 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.531320 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 22:22:28.532285 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.532268 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s7t6h" Jul 10 22:22:28.534348 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.534328 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 22:22:28.534591 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.534572 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wnjc6\"" Jul 10 22:22:28.534668 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.534633 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 22:22:28.534917 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.534896 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 22:22:28.541973 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.541917 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8s7sv"] Jul 10 22:22:28.542771 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.542749 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s7t6h"] Jul 10 22:22:28.575786 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.575754 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc1d3dd-e770-4942-b849-c31dd37c52fa-config-volume\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.575933 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.575799 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shdfp\" (UniqueName: \"kubernetes.io/projected/4fc1d3dd-e770-4942-b849-c31dd37c52fa-kube-api-access-shdfp\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.575933 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.575817 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g9dz\" (UniqueName: \"kubernetes.io/projected/8ad3159f-50bb-4fc7-af39-e85508a1575d-kube-api-access-4g9dz\") pod \"ingress-canary-s7t6h\" (UID: \"8ad3159f-50bb-4fc7-af39-e85508a1575d\") " pod="openshift-ingress-canary/ingress-canary-s7t6h" Jul 10 22:22:28.575933 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.575869 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fc1d3dd-e770-4942-b849-c31dd37c52fa-tmp-dir\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.575933 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.575890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ad3159f-50bb-4fc7-af39-e85508a1575d-cert\") pod \"ingress-canary-s7t6h\" (UID: \"8ad3159f-50bb-4fc7-af39-e85508a1575d\") " pod="openshift-ingress-canary/ingress-canary-s7t6h" Jul 10 22:22:28.576093 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.575951 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.576093 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.575970 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-ca-trust-extracted\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.576093 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.575985 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-installation-pull-secrets\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.576093 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.576010 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4t2t\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-kube-api-access-p4t2t\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.576093 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.576034 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-bound-sa-token\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.576093 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.576086 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-image-registry-private-configuration\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.576293 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.576121 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.576293 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.576156 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-certificates\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.576293 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.576175 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-trusted-ca\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.676524 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.676479 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fc1d3dd-e770-4942-b849-c31dd37c52fa-tmp-dir\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.676700 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.676533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ad3159f-50bb-4fc7-af39-e85508a1575d-cert\") pod \"ingress-canary-s7t6h\" (UID: \"8ad3159f-50bb-4fc7-af39-e85508a1575d\") " pod="openshift-ingress-canary/ingress-canary-s7t6h" Jul 10 22:22:28.676700 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.676583 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.676700 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.676608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-ca-trust-extracted\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.676855 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:28.676709 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 22:22:28.676855 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:28.676733 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db6cd9c68-dft8m: secret "image-registry-tls" not found Jul 10 22:22:28.676855 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.676782 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fc1d3dd-e770-4942-b849-c31dd37c52fa-tmp-dir\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.676855 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:28.676804 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls podName:a41d9dd7-0403-43e9-9ea9-11e18f8d32a4 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:29.176784661 +0000 UTC m=+37.025066528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls") pod "image-registry-7db6cd9c68-dft8m" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4") : secret "image-registry-tls" not found Jul 10 22:22:28.676855 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.676827 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-installation-pull-secrets\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.677120 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.676866 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4t2t\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-kube-api-access-p4t2t\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.677120 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.676890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-bound-sa-token\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.677120 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.676915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-image-registry-private-configuration\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.677120 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.676942 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.677120 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.676978 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-certificates\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.677120 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.677002 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-trusted-ca\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.677120 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.677027 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-ca-trust-extracted\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.677120 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.677034 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc1d3dd-e770-4942-b849-c31dd37c52fa-config-volume\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.677120 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.677086 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shdfp\" (UniqueName: \"kubernetes.io/projected/4fc1d3dd-e770-4942-b849-c31dd37c52fa-kube-api-access-shdfp\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.677120 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.677111 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g9dz\" (UniqueName: \"kubernetes.io/projected/8ad3159f-50bb-4fc7-af39-e85508a1575d-kube-api-access-4g9dz\") pod \"ingress-canary-s7t6h\" (UID: \"8ad3159f-50bb-4fc7-af39-e85508a1575d\") " pod="openshift-ingress-canary/ingress-canary-s7t6h" Jul 10 22:22:28.677623 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.677519 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4fc1d3dd-e770-4942-b849-c31dd37c52fa-config-volume\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.677623 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:28.677598 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:22:28.677731 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:28.677653 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls podName:4fc1d3dd-e770-4942-b849-c31dd37c52fa nodeName:}" failed. No retries permitted until 2026-07-10 22:22:29.177638591 +0000 UTC m=+37.025920455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls") pod "dns-default-8s7sv" (UID: "4fc1d3dd-e770-4942-b849-c31dd37c52fa") : secret "dns-default-metrics-tls" not found Jul 10 22:22:28.678360 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.678189 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-certificates\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.678531 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.678510 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-trusted-ca\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.681580 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.681558 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-image-registry-private-configuration\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.681681 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.681578 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-installation-pull-secrets\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.681681 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.681599 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8ad3159f-50bb-4fc7-af39-e85508a1575d-cert\") pod \"ingress-canary-s7t6h\" (UID: \"8ad3159f-50bb-4fc7-af39-e85508a1575d\") " pod="openshift-ingress-canary/ingress-canary-s7t6h" Jul 10 22:22:28.688016 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.687980 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g9dz\" (UniqueName: \"kubernetes.io/projected/8ad3159f-50bb-4fc7-af39-e85508a1575d-kube-api-access-4g9dz\") pod \"ingress-canary-s7t6h\" (UID: \"8ad3159f-50bb-4fc7-af39-e85508a1575d\") " pod="openshift-ingress-canary/ingress-canary-s7t6h" Jul 10 22:22:28.688782 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.688708 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shdfp\" (UniqueName: \"kubernetes.io/projected/4fc1d3dd-e770-4942-b849-c31dd37c52fa-kube-api-access-shdfp\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:28.689989 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.689968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-bound-sa-token\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.690775 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.690753 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4t2t\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-kube-api-access-p4t2t\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:28.700878 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.700854 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t2tv_8e7057db-a92a-460c-9cbc-7f9815afa729/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:28.719959 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.719937 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:28.722551 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.722316 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qbckj\"" Jul 10 22:22:28.722551 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.722316 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 22:22:28.784130 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.784087 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t2tv_8e7057db-a92a-460c-9cbc-7f9815afa729/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:28.846333 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.846136 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-s7t6h" Jul 10 22:22:28.989822 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:28.989795 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-s7t6h"] Jul 10 22:22:28.993306 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:22:28.993279 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad3159f_50bb_4fc7_af39_e85508a1575d.slice/crio-2f2819eae763f4f5d4e41dbc3cdecd9ef381711c7e800c55d1fd2b4692004083 WatchSource:0}: Error finding container 2f2819eae763f4f5d4e41dbc3cdecd9ef381711c7e800c55d1fd2b4692004083: Status 404 returned error can't find the container with id 2f2819eae763f4f5d4e41dbc3cdecd9ef381711c7e800c55d1fd2b4692004083 Jul 10 22:22:29.180208 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:29.180159 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:29.180379 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:29.180225 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:29.180379 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:29.180327 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 22:22:29.180379 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:29.180351 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db6cd9c68-dft8m: secret "image-registry-tls" not found Jul 10 22:22:29.180546 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:29.180416 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls podName:a41d9dd7-0403-43e9-9ea9-11e18f8d32a4 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:30.180392814 +0000 UTC m=+38.028674669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls") pod "image-registry-7db6cd9c68-dft8m" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4") : secret "image-registry-tls" not found Jul 10 22:22:29.180546 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:29.180325 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:22:29.180546 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:29.180478 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls podName:4fc1d3dd-e770-4942-b849-c31dd37c52fa nodeName:}" failed. No retries permitted until 2026-07-10 22:22:30.180466889 +0000 UTC m=+38.028748742 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls") pod "dns-default-8s7sv" (UID: "4fc1d3dd-e770-4942-b849-c31dd37c52fa") : secret "dns-default-metrics-tls" not found Jul 10 22:22:29.719408 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:29.719358 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:29.719587 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:29.719358 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:29.722630 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:29.722605 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 22:22:29.723022 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:29.722633 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 22:22:29.723022 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:29.722605 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ts5wt\"" Jul 10 22:22:29.723306 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:29.723289 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 22:22:29.910885 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:29.910846 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s7t6h" event={"ID":"8ad3159f-50bb-4fc7-af39-e85508a1575d","Type":"ContainerStarted","Data":"2f2819eae763f4f5d4e41dbc3cdecd9ef381711c7e800c55d1fd2b4692004083"} Jul 10 22:22:29.914076 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:29.914041 2559 generic.go:356] "Generic (PLEG): container finished" podID="6171ceb6-38d3-46f3-a504-42094bd28cde" containerID="bf448395a39b652cd8ffca45017f684b8b93cda41f4030270d6a9d934077bf73" exitCode=0 Jul 10 22:22:29.914277 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:29.914091 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" event={"ID":"6171ceb6-38d3-46f3-a504-42094bd28cde","Type":"ContainerDied","Data":"bf448395a39b652cd8ffca45017f684b8b93cda41f4030270d6a9d934077bf73"} Jul 10 22:22:30.187537 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:30.187503 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:30.187753 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:30.187570 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:30.187753 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:30.187698 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 22:22:30.187753 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:30.187724 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db6cd9c68-dft8m: secret "image-registry-tls" not found Jul 10 22:22:30.187753 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:30.187740 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:22:30.188030 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:30.187798 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls podName:a41d9dd7-0403-43e9-9ea9-11e18f8d32a4 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:32.187776476 +0000 UTC m=+40.036058337 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls") pod "image-registry-7db6cd9c68-dft8m" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4") : secret "image-registry-tls" not found Jul 10 22:22:30.188030 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:30.187818 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls podName:4fc1d3dd-e770-4942-b849-c31dd37c52fa nodeName:}" failed. No retries permitted until 2026-07-10 22:22:32.187809582 +0000 UTC m=+40.036091436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls") pod "dns-default-8s7sv" (UID: "4fc1d3dd-e770-4942-b849-c31dd37c52fa") : secret "dns-default-metrics-tls" not found Jul 10 22:22:30.920541 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:30.920503 2559 generic.go:356] "Generic (PLEG): container finished" podID="6171ceb6-38d3-46f3-a504-42094bd28cde" containerID="72c61ccc4e45ebf1c474df83d790c9dd228c696ebca5aa5d2aee790f5ecbad8b" exitCode=0 Jul 10 22:22:30.920908 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:30.920565 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" event={"ID":"6171ceb6-38d3-46f3-a504-42094bd28cde","Type":"ContainerDied","Data":"72c61ccc4e45ebf1c474df83d790c9dd228c696ebca5aa5d2aee790f5ecbad8b"} Jul 10 22:22:31.924989 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:31.924762 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-s7t6h" event={"ID":"8ad3159f-50bb-4fc7-af39-e85508a1575d","Type":"ContainerStarted","Data":"608e8e8d21c5fc4bc14dcb51d93ea793a048a28262fb64683a5d2d5ea68c87b0"} Jul 10 22:22:31.928555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:31.928527 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" event={"ID":"6171ceb6-38d3-46f3-a504-42094bd28cde","Type":"ContainerStarted","Data":"99fdc2e98492cc8df30b833b67f7e00d969d5de894897718ca2672f708c290cd"} Jul 10 22:22:31.946724 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:31.946387 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-s7t6h" podStartSLOduration=1.137245053 podStartE2EDuration="3.946371715s" totalImagesPullingTime="2.809126662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:31.946183105 +0000 UTC m=+39.794464982" watchObservedRunningTime="2026-07-10 22:22:31.946371715 +0000 UTC m=+39.794653592" Jul 10 22:22:31.973119 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:31.973087 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t2tv_8e7057db-a92a-460c-9cbc-7f9815afa729/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:32.073725 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:32.073648 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t2tv_8e7057db-a92a-460c-9cbc-7f9815afa729/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:32.172974 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:32.172940 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t2tv_8e7057db-a92a-460c-9cbc-7f9815afa729/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:32.201131 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:32.201104 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:32.201240 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:32.201154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:32.201314 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:32.201253 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:22:32.201314 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:32.201262 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 22:22:32.201314 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:32.201278 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db6cd9c68-dft8m: secret "image-registry-tls" not found Jul 10 22:22:32.201314 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:32.201306 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls podName:4fc1d3dd-e770-4942-b849-c31dd37c52fa nodeName:}" failed. No retries permitted until 2026-07-10 22:22:36.201291811 +0000 UTC m=+44.049573671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls") pod "dns-default-8s7sv" (UID: "4fc1d3dd-e770-4942-b849-c31dd37c52fa") : secret "dns-default-metrics-tls" not found Jul 10 22:22:32.201473 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:32.201329 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls podName:a41d9dd7-0403-43e9-9ea9-11e18f8d32a4 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:36.201312378 +0000 UTC m=+44.049594251 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls") pod "image-registry-7db6cd9c68-dft8m" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4") : secret "image-registry-tls" not found Jul 10 22:22:32.273998 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:32.273977 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t2tv_8e7057db-a92a-460c-9cbc-7f9815afa729/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:32.323789 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:32.323719 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t2tv_8e7057db-a92a-460c-9cbc-7f9815afa729/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:34.288448 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:34.288392 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t2tv_8e7057db-a92a-460c-9cbc-7f9815afa729/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:34.343006 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:34.342977 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t2tv_8e7057db-a92a-460c-9cbc-7f9815afa729/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:35.829985 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:35.829952 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9t2tv_8e7057db-a92a-460c-9cbc-7f9815afa729/konnectivity-agent/0.log" limit=1024 Jul 10 22:22:36.229066 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:36.229026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:36.229219 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:36.229091 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:36.229219 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:36.229181 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:22:36.229310 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:36.229245 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls podName:4fc1d3dd-e770-4942-b849-c31dd37c52fa nodeName:}" failed. No retries permitted until 2026-07-10 22:22:44.229231265 +0000 UTC m=+52.077513122 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls") pod "dns-default-8s7sv" (UID: "4fc1d3dd-e770-4942-b849-c31dd37c52fa") : secret "dns-default-metrics-tls" not found Jul 10 22:22:36.229310 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:36.229184 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 22:22:36.229310 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:36.229278 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db6cd9c68-dft8m: secret "image-registry-tls" not found Jul 10 22:22:36.229310 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:36.229300 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls podName:a41d9dd7-0403-43e9-9ea9-11e18f8d32a4 nodeName:}" failed. No retries permitted until 2026-07-10 22:22:44.229293719 +0000 UTC m=+52.077575573 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls") pod "image-registry-7db6cd9c68-dft8m" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4") : secret "image-registry-tls" not found Jul 10 22:22:44.286919 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:44.286885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:22:44.287389 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:44.286935 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:22:44.287389 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:44.287022 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:22:44.287389 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:44.287039 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 22:22:44.287389 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:44.287057 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db6cd9c68-dft8m: secret "image-registry-tls" not found Jul 10 22:22:44.287389 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:44.287070 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls podName:4fc1d3dd-e770-4942-b849-c31dd37c52fa nodeName:}" failed. No retries permitted until 2026-07-10 22:23:00.287057531 +0000 UTC m=+68.135339385 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls") pod "dns-default-8s7sv" (UID: "4fc1d3dd-e770-4942-b849-c31dd37c52fa") : secret "dns-default-metrics-tls" not found Jul 10 22:22:44.287389 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:44.287109 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls podName:a41d9dd7-0403-43e9-9ea9-11e18f8d32a4 nodeName:}" failed. No retries permitted until 2026-07-10 22:23:00.287096081 +0000 UTC m=+68.135377934 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls") pod "image-registry-7db6cd9c68-dft8m" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4") : secret "image-registry-tls" not found Jul 10 22:22:47.194532 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.194474 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nfvl2" podStartSLOduration=24.413435136 podStartE2EDuration="54.194458828s" totalImagesPullingTime="29.781023692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:21:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:31.968899221 +0000 UTC m=+39.817181110" watchObservedRunningTime="2026-07-10 22:22:47.194458828 +0000 UTC m=+55.042740701" Jul 10 22:22:47.195390 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.195368 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-k8jhp"] Jul 10 22:22:47.214303 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.210958 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.214516 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.214492 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 10 22:22:47.215003 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.214805 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-krhf7\"" Jul 10 22:22:47.215395 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.215380 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 22:22:47.216207 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.215891 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-k8jhp"] Jul 10 22:22:47.216337 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.215992 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 22:22:47.216477 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.216052 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 10 22:22:47.226554 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.226533 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 10 22:22:47.289464 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.289392 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw"] Jul 10 22:22:47.301022 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.301000 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk"] Jul 10 22:22:47.301139 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.301124 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" Jul 10 22:22:47.303478 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.303462 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 10 22:22:47.304263 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.304245 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 10 22:22:47.304397 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.304373 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 10 22:22:47.304522 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.304399 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-d7dxq\"" Jul 10 22:22:47.304522 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.304309 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:22:47.309526 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.309505 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-snapshots\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.309640 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.309542 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.309640 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.309583 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-tmp\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.309640 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.309606 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.309796 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.309641 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dpjh\" (UniqueName: \"kubernetes.io/projected/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-kube-api-access-7dpjh\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.309796 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.309749 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-serving-cert\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.320004 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.319984 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-9g2cr"] Jul 10 22:22:47.320118 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.320103 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:22:47.322229 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.322194 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:22:47.322379 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.322358 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 10 22:22:47.322379 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.322371 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-md7xf\"" Jul 10 22:22:47.322558 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.322419 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 10 22:22:47.339531 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.339508 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw"] Jul 10 22:22:47.339616 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.339563 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk"] Jul 10 22:22:47.339616 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.339595 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-9g2cr" Jul 10 22:22:47.339680 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.339641 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-9g2cr"] Jul 10 22:22:47.341842 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.341824 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xsp5r\"" Jul 10 22:22:47.391095 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.391072 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p"] Jul 10 22:22:47.410617 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.410597 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.410716 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.410627 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zrq2\" (UniqueName: \"kubernetes.io/projected/10b1ab46-05e5-413a-9c4e-5abde8f71eed-kube-api-access-4zrq2\") pod \"service-ca-operator-7695fc6c8c-g9djw\" (UID: \"10b1ab46-05e5-413a-9c4e-5abde8f71eed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" Jul 10 22:22:47.410716 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.410647 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cpfzk\" (UID: \"1604ab06-423e-4153-8138-6087e8f42f9a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:22:47.410716 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.410680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10b1ab46-05e5-413a-9c4e-5abde8f71eed-config\") pod \"service-ca-operator-7695fc6c8c-g9djw\" (UID: \"10b1ab46-05e5-413a-9c4e-5abde8f71eed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" Jul 10 22:22:47.410828 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.410716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-tmp\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.410828 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.410751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.410896 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.410866 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10b1ab46-05e5-413a-9c4e-5abde8f71eed-serving-cert\") pod \"service-ca-operator-7695fc6c8c-g9djw\" (UID: \"10b1ab46-05e5-413a-9c4e-5abde8f71eed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" Jul 10 22:22:47.410932 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.410916 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dpjh\" (UniqueName: \"kubernetes.io/projected/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-kube-api-access-7dpjh\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.410969 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.410957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-serving-cert\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.411009 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.410974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz4mb\" (UniqueName: \"kubernetes.io/projected/1604ab06-423e-4153-8138-6087e8f42f9a-kube-api-access-mz4mb\") pod \"cluster-samples-operator-65db858894-cpfzk\" (UID: \"1604ab06-423e-4153-8138-6087e8f42f9a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:22:47.411057 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.411043 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-snapshots\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.411112 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.411080 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h94c\" (UniqueName: \"kubernetes.io/projected/c0adccc9-2104-4287-8d3c-cd86ffb3867c-kube-api-access-2h94c\") pod \"network-check-source-869dcfcdcd-9g2cr\" (UID: \"c0adccc9-2104-4287-8d3c-cd86ffb3867c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-9g2cr" Jul 10 22:22:47.411164 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.411114 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-tmp\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.411260 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.411243 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.411551 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.411532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.411627 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.411576 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-snapshots\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.412322 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.412304 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p"] Jul 10 22:22:47.412402 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.412380 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" Jul 10 22:22:47.414475 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.414458 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 10 22:22:47.414592 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.414568 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 10 22:22:47.414688 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.414608 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4tb56\"" Jul 10 22:22:47.414688 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.414634 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 10 22:22:47.415035 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.415010 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:22:47.415223 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.415207 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-serving-cert\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.418511 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.418490 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dpjh\" (UniqueName: \"kubernetes.io/projected/64cfd78b-f6a5-4da0-ab73-dd6928c55dae-kube-api-access-7dpjh\") pod \"insights-operator-68d9dd94cf-k8jhp\" (UID: \"64cfd78b-f6a5-4da0-ab73-dd6928c55dae\") " pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.511663 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.511640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zrq2\" (UniqueName: \"kubernetes.io/projected/10b1ab46-05e5-413a-9c4e-5abde8f71eed-kube-api-access-4zrq2\") pod \"service-ca-operator-7695fc6c8c-g9djw\" (UID: \"10b1ab46-05e5-413a-9c4e-5abde8f71eed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" Jul 10 22:22:47.511767 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.511671 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr9xz\" (UniqueName: \"kubernetes.io/projected/2be24f5f-4532-4c5f-bebf-5c5b76a699cd-kube-api-access-qr9xz\") pod \"kube-storage-version-migrator-operator-799cb7f958-mtt7p\" (UID: \"2be24f5f-4532-4c5f-bebf-5c5b76a699cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" Jul 10 22:22:47.511767 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.511712 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2be24f5f-4532-4c5f-bebf-5c5b76a699cd-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-mtt7p\" (UID: \"2be24f5f-4532-4c5f-bebf-5c5b76a699cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" Jul 10 22:22:47.511767 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.511729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h94c\" (UniqueName: \"kubernetes.io/projected/c0adccc9-2104-4287-8d3c-cd86ffb3867c-kube-api-access-2h94c\") pod \"network-check-source-869dcfcdcd-9g2cr\" (UID: \"c0adccc9-2104-4287-8d3c-cd86ffb3867c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-9g2cr" Jul 10 22:22:47.511877 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.511771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cpfzk\" (UID: \"1604ab06-423e-4153-8138-6087e8f42f9a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:22:47.511877 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.511791 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10b1ab46-05e5-413a-9c4e-5abde8f71eed-config\") pod \"service-ca-operator-7695fc6c8c-g9djw\" (UID: \"10b1ab46-05e5-413a-9c4e-5abde8f71eed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" Jul 10 22:22:47.511877 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.511808 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10b1ab46-05e5-413a-9c4e-5abde8f71eed-serving-cert\") pod \"service-ca-operator-7695fc6c8c-g9djw\" (UID: \"10b1ab46-05e5-413a-9c4e-5abde8f71eed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" Jul 10 22:22:47.511877 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.511855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz4mb\" (UniqueName: \"kubernetes.io/projected/1604ab06-423e-4153-8138-6087e8f42f9a-kube-api-access-mz4mb\") pod \"cluster-samples-operator-65db858894-cpfzk\" (UID: \"1604ab06-423e-4153-8138-6087e8f42f9a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:22:47.512038 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:47.511903 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 22:22:47.512038 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:47.511970 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls podName:1604ab06-423e-4153-8138-6087e8f42f9a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:48.011951662 +0000 UTC m=+55.860233526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls") pod "cluster-samples-operator-65db858894-cpfzk" (UID: "1604ab06-423e-4153-8138-6087e8f42f9a") : secret "samples-operator-tls" not found Jul 10 22:22:47.512038 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.511964 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2be24f5f-4532-4c5f-bebf-5c5b76a699cd-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-mtt7p\" (UID: \"2be24f5f-4532-4c5f-bebf-5c5b76a699cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" Jul 10 22:22:47.512332 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.512310 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10b1ab46-05e5-413a-9c4e-5abde8f71eed-config\") pod \"service-ca-operator-7695fc6c8c-g9djw\" (UID: \"10b1ab46-05e5-413a-9c4e-5abde8f71eed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" Jul 10 22:22:47.513791 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.513771 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/10b1ab46-05e5-413a-9c4e-5abde8f71eed-serving-cert\") pod \"service-ca-operator-7695fc6c8c-g9djw\" (UID: \"10b1ab46-05e5-413a-9c4e-5abde8f71eed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" Jul 10 22:22:47.523421 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.523400 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz4mb\" (UniqueName: \"kubernetes.io/projected/1604ab06-423e-4153-8138-6087e8f42f9a-kube-api-access-mz4mb\") pod \"cluster-samples-operator-65db858894-cpfzk\" (UID: \"1604ab06-423e-4153-8138-6087e8f42f9a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:22:47.523617 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.523600 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zrq2\" (UniqueName: \"kubernetes.io/projected/10b1ab46-05e5-413a-9c4e-5abde8f71eed-kube-api-access-4zrq2\") pod \"service-ca-operator-7695fc6c8c-g9djw\" (UID: \"10b1ab46-05e5-413a-9c4e-5abde8f71eed\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" Jul 10 22:22:47.523676 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.523633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h94c\" (UniqueName: \"kubernetes.io/projected/c0adccc9-2104-4287-8d3c-cd86ffb3867c-kube-api-access-2h94c\") pod \"network-check-source-869dcfcdcd-9g2cr\" (UID: \"c0adccc9-2104-4287-8d3c-cd86ffb3867c\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-9g2cr" Jul 10 22:22:47.533432 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.533416 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" Jul 10 22:22:47.610494 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.610462 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" Jul 10 22:22:47.612776 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.612754 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2be24f5f-4532-4c5f-bebf-5c5b76a699cd-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-mtt7p\" (UID: \"2be24f5f-4532-4c5f-bebf-5c5b76a699cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" Jul 10 22:22:47.612857 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.612815 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr9xz\" (UniqueName: \"kubernetes.io/projected/2be24f5f-4532-4c5f-bebf-5c5b76a699cd-kube-api-access-qr9xz\") pod \"kube-storage-version-migrator-operator-799cb7f958-mtt7p\" (UID: \"2be24f5f-4532-4c5f-bebf-5c5b76a699cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" Jul 10 22:22:47.612894 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.612862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2be24f5f-4532-4c5f-bebf-5c5b76a699cd-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-mtt7p\" (UID: \"2be24f5f-4532-4c5f-bebf-5c5b76a699cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" Jul 10 22:22:47.613315 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.613293 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2be24f5f-4532-4c5f-bebf-5c5b76a699cd-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-mtt7p\" (UID: \"2be24f5f-4532-4c5f-bebf-5c5b76a699cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" Jul 10 22:22:47.615050 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.614897 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2be24f5f-4532-4c5f-bebf-5c5b76a699cd-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-mtt7p\" (UID: \"2be24f5f-4532-4c5f-bebf-5c5b76a699cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" Jul 10 22:22:47.623268 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.623246 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr9xz\" (UniqueName: \"kubernetes.io/projected/2be24f5f-4532-4c5f-bebf-5c5b76a699cd-kube-api-access-qr9xz\") pod \"kube-storage-version-migrator-operator-799cb7f958-mtt7p\" (UID: \"2be24f5f-4532-4c5f-bebf-5c5b76a699cd\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" Jul 10 22:22:47.647732 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.647709 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-9g2cr" Jul 10 22:22:47.682270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.682238 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-k8jhp"] Jul 10 22:22:47.689808 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:22:47.689764 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cfd78b_f6a5_4da0_ab73_dd6928c55dae.slice/crio-77ebf3c1805ff4fe5c171b94375aa1405bf3fa19e011e2d7f72977401b231744 WatchSource:0}: Error finding container 77ebf3c1805ff4fe5c171b94375aa1405bf3fa19e011e2d7f72977401b231744: Status 404 returned error can't find the container with id 77ebf3c1805ff4fe5c171b94375aa1405bf3fa19e011e2d7f72977401b231744 Jul 10 22:22:47.728131 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.728066 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" Jul 10 22:22:47.764951 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.764882 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw"] Jul 10 22:22:47.768239 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:22:47.768211 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b1ab46_05e5_413a_9c4e_5abde8f71eed.slice/crio-117d09128a1ebf4be9d22b2695c636212ddce0aff4eef83ad8a3920dad2e7a4b WatchSource:0}: Error finding container 117d09128a1ebf4be9d22b2695c636212ddce0aff4eef83ad8a3920dad2e7a4b: Status 404 returned error can't find the container with id 117d09128a1ebf4be9d22b2695c636212ddce0aff4eef83ad8a3920dad2e7a4b Jul 10 22:22:47.811086 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.811057 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-9g2cr"] Jul 10 22:22:47.814196 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:22:47.814170 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0adccc9_2104_4287_8d3c_cd86ffb3867c.slice/crio-22adac70c3bb2dba8ae2ac05a443bd457dc2ae9d73214d5acbc902af5a39d39e WatchSource:0}: Error finding container 22adac70c3bb2dba8ae2ac05a443bd457dc2ae9d73214d5acbc902af5a39d39e: Status 404 returned error can't find the container with id 22adac70c3bb2dba8ae2ac05a443bd457dc2ae9d73214d5acbc902af5a39d39e Jul 10 22:22:47.861933 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.861907 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p"] Jul 10 22:22:47.864760 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:22:47.864736 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be24f5f_4532_4c5f_bebf_5c5b76a699cd.slice/crio-38ceb541373439b0d96b0332dca51844ef69b0484a793db6a9a147c133c856f3 WatchSource:0}: Error finding container 38ceb541373439b0d96b0332dca51844ef69b0484a793db6a9a147c133c856f3: Status 404 returned error can't find the container with id 38ceb541373439b0d96b0332dca51844ef69b0484a793db6a9a147c133c856f3 Jul 10 22:22:47.967472 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.967425 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" event={"ID":"2be24f5f-4532-4c5f-bebf-5c5b76a699cd","Type":"ContainerStarted","Data":"38ceb541373439b0d96b0332dca51844ef69b0484a793db6a9a147c133c856f3"} Jul 10 22:22:47.968410 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.968380 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" event={"ID":"64cfd78b-f6a5-4da0-ab73-dd6928c55dae","Type":"ContainerStarted","Data":"77ebf3c1805ff4fe5c171b94375aa1405bf3fa19e011e2d7f72977401b231744"} Jul 10 22:22:47.969369 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.969348 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-9g2cr" event={"ID":"c0adccc9-2104-4287-8d3c-cd86ffb3867c","Type":"ContainerStarted","Data":"22adac70c3bb2dba8ae2ac05a443bd457dc2ae9d73214d5acbc902af5a39d39e"} Jul 10 22:22:47.970350 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:47.970332 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" event={"ID":"10b1ab46-05e5-413a-9c4e-5abde8f71eed","Type":"ContainerStarted","Data":"117d09128a1ebf4be9d22b2695c636212ddce0aff4eef83ad8a3920dad2e7a4b"} Jul 10 22:22:48.015928 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:48.015901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cpfzk\" (UID: \"1604ab06-423e-4153-8138-6087e8f42f9a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:22:48.016060 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:48.016044 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 22:22:48.016110 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:48.016099 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls podName:1604ab06-423e-4153-8138-6087e8f42f9a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:49.016086258 +0000 UTC m=+56.864368111 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls") pod "cluster-samples-operator-65db858894-cpfzk" (UID: "1604ab06-423e-4153-8138-6087e8f42f9a") : secret "samples-operator-tls" not found Jul 10 22:22:49.024507 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:49.024454 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cpfzk\" (UID: \"1604ab06-423e-4153-8138-6087e8f42f9a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:22:49.024961 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:49.024685 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 22:22:49.024961 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:49.024751 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls podName:1604ab06-423e-4153-8138-6087e8f42f9a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:51.024731803 +0000 UTC m=+58.873013672 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls") pod "cluster-samples-operator-65db858894-cpfzk" (UID: "1604ab06-423e-4153-8138-6087e8f42f9a") : secret "samples-operator-tls" not found Jul 10 22:22:51.042476 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:51.042419 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cpfzk\" (UID: \"1604ab06-423e-4153-8138-6087e8f42f9a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:22:51.043010 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:51.042595 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 22:22:51.043010 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:51.042668 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls podName:1604ab06-423e-4153-8138-6087e8f42f9a nodeName:}" failed. No retries permitted until 2026-07-10 22:22:55.042651039 +0000 UTC m=+62.890932893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls") pod "cluster-samples-operator-65db858894-cpfzk" (UID: "1604ab06-423e-4153-8138-6087e8f42f9a") : secret "samples-operator-tls" not found Jul 10 22:22:52.902615 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:52.902583 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tkx48" Jul 10 22:22:52.991826 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:52.991786 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" event={"ID":"64cfd78b-f6a5-4da0-ab73-dd6928c55dae","Type":"ContainerStarted","Data":"8e24d91f59755d0cb41820ce6bd08a258328c86d9b703b8ee8f6e2419f15e65c"} Jul 10 22:22:52.993681 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:52.993648 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-9g2cr" event={"ID":"c0adccc9-2104-4287-8d3c-cd86ffb3867c","Type":"ContainerStarted","Data":"604bf8cc0e402c5b39308f5ebb64f6597c4de1489db0e9a603f0cff97749aa59"} Jul 10 22:22:52.995828 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:52.995799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" event={"ID":"10b1ab46-05e5-413a-9c4e-5abde8f71eed","Type":"ContainerStarted","Data":"b9237d1cdbd1c5870ded7d4fac1bb4defb65afbe8f68255d704718887c262eac"} Jul 10 22:22:52.997631 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:52.997610 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" event={"ID":"2be24f5f-4532-4c5f-bebf-5c5b76a699cd","Type":"ContainerStarted","Data":"c8bf53378d26205cd446798ad67bfeb571d21b887f75bc2f06030037d13d7355"} Jul 10 22:22:53.110019 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:53.109946 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" podStartSLOduration=1.880749971 podStartE2EDuration="6.109928093s" totalImagesPullingTime="4.229178122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:53.060917995 +0000 UTC m=+60.909199870" watchObservedRunningTime="2026-07-10 22:22:53.109928093 +0000 UTC m=+60.958209967" Jul 10 22:22:53.130169 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:53.129955 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" podStartSLOduration=1.974613217 podStartE2EDuration="6.129937268s" totalImagesPullingTime="4.155324051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:53.110560054 +0000 UTC m=+60.958841931" watchObservedRunningTime="2026-07-10 22:22:53.129937268 +0000 UTC m=+60.978219147" Jul 10 22:22:53.130583 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:53.130552 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-9g2cr" podStartSLOduration=2.01188855 podStartE2EDuration="6.130540354s" totalImagesPullingTime="4.118651804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:53.128899406 +0000 UTC m=+60.977181279" watchObservedRunningTime="2026-07-10 22:22:53.130540354 +0000 UTC m=+60.978822232" Jul 10 22:22:53.150275 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:53.150216 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" podStartSLOduration=2.093444362 podStartE2EDuration="6.150196813s" totalImagesPullingTime="4.056752451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:53.1493147 +0000 UTC m=+60.997596577" watchObservedRunningTime="2026-07-10 22:22:53.150196813 +0000 UTC m=+60.998478696" Jul 10 22:22:53.969747 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:53.969033 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kcnnj"] Jul 10 22:22:53.993263 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:53.993222 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kcnnj"] Jul 10 22:22:53.993418 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:53.993361 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:53.995841 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:53.995813 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 22:22:53.995964 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:53.995917 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 22:22:53.995964 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:53.995841 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bhc4w\"" Jul 10 22:22:54.067219 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.067187 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc1678f8-4523-4495-90d5-a35bd0fa292d-crio-socket\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.067692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.067669 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1678f8-4523-4495-90d5-a35bd0fa292d-data-volume\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.067812 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.067706 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.067812 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.067770 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrjnx\" (UniqueName: \"kubernetes.io/projected/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-api-access-lrjnx\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.067940 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.067838 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.169166 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.169127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1678f8-4523-4495-90d5-a35bd0fa292d-data-volume\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.169166 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.169168 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.169397 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:54.169245 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 22:22:54.169397 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.169293 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrjnx\" (UniqueName: \"kubernetes.io/projected/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-api-access-lrjnx\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.169397 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:54.169327 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm podName:fc1678f8-4523-4495-90d5-a35bd0fa292d nodeName:}" failed. No retries permitted until 2026-07-10 22:22:54.669305923 +0000 UTC m=+62.517587803 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kcnnj" (UID: "fc1678f8-4523-4495-90d5-a35bd0fa292d") : configmap "kube-rbac-proxy" not found Jul 10 22:22:54.169397 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.169368 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.169620 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.169410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc1678f8-4523-4495-90d5-a35bd0fa292d-crio-socket\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.169620 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:54.169516 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 22:22:54.169620 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.169530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fc1678f8-4523-4495-90d5-a35bd0fa292d-data-volume\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.169620 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.169540 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fc1678f8-4523-4495-90d5-a35bd0fa292d-crio-socket\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.169620 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:54.169579 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls podName:fc1678f8-4523-4495-90d5-a35bd0fa292d nodeName:}" failed. No retries permitted until 2026-07-10 22:22:54.669563252 +0000 UTC m=+62.517845109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kcnnj" (UID: "fc1678f8-4523-4495-90d5-a35bd0fa292d") : secret "insights-runtime-extractor-tls" not found Jul 10 22:22:54.183395 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.183372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrjnx\" (UniqueName: \"kubernetes.io/projected/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-api-access-lrjnx\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.674654 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.674619 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.674814 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:54.674735 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:54.674814 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:54.674784 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 22:22:54.674814 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:54.674800 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 22:22:54.674917 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:54.674851 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm podName:fc1678f8-4523-4495-90d5-a35bd0fa292d nodeName:}" failed. No retries permitted until 2026-07-10 22:22:55.674836657 +0000 UTC m=+63.523118529 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kcnnj" (UID: "fc1678f8-4523-4495-90d5-a35bd0fa292d") : configmap "kube-rbac-proxy" not found Jul 10 22:22:54.674917 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:54.674865 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls podName:fc1678f8-4523-4495-90d5-a35bd0fa292d nodeName:}" failed. No retries permitted until 2026-07-10 22:22:55.674859865 +0000 UTC m=+63.523141719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kcnnj" (UID: "fc1678f8-4523-4495-90d5-a35bd0fa292d") : secret "insights-runtime-extractor-tls" not found Jul 10 22:22:55.078021 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:55.077940 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cpfzk\" (UID: \"1604ab06-423e-4153-8138-6087e8f42f9a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:22:55.078372 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:55.078068 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 10 22:22:55.078372 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:55.078140 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls podName:1604ab06-423e-4153-8138-6087e8f42f9a nodeName:}" failed. No retries permitted until 2026-07-10 22:23:03.078116651 +0000 UTC m=+70.926398519 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls") pod "cluster-samples-operator-65db858894-cpfzk" (UID: "1604ab06-423e-4153-8138-6087e8f42f9a") : secret "samples-operator-tls" not found Jul 10 22:22:55.683731 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:55.683699 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:55.683916 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:55.683759 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:55.683916 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:55.683831 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 22:22:55.683916 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:55.683891 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 22:22:55.683916 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:55.683902 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm podName:fc1678f8-4523-4495-90d5-a35bd0fa292d nodeName:}" failed. No retries permitted until 2026-07-10 22:22:57.683887255 +0000 UTC m=+65.532169132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kcnnj" (UID: "fc1678f8-4523-4495-90d5-a35bd0fa292d") : configmap "kube-rbac-proxy" not found Jul 10 22:22:55.684070 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:55.683927 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls podName:fc1678f8-4523-4495-90d5-a35bd0fa292d nodeName:}" failed. No retries permitted until 2026-07-10 22:22:57.683916786 +0000 UTC m=+65.532198639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kcnnj" (UID: "fc1678f8-4523-4495-90d5-a35bd0fa292d") : secret "insights-runtime-extractor-tls" not found Jul 10 22:22:56.176734 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.176702 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m5kxh_2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7/dns-node-resolver/0.log" Jul 10 22:22:56.454891 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.454816 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-spd8d"] Jul 10 22:22:56.457488 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.457468 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" Jul 10 22:22:56.459732 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.459713 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 10 22:22:56.459732 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.459726 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 10 22:22:56.460632 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.460618 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-2qq85\"" Jul 10 22:22:56.460702 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.460618 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 10 22:22:56.460702 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.460662 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 10 22:22:56.465258 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.465234 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-spd8d"] Jul 10 22:22:56.593547 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.593515 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f09d305a-458b-4609-8a90-0a1e2ade0498-signing-cabundle\") pod \"service-ca-79f68b9f4f-spd8d\" (UID: \"f09d305a-458b-4609-8a90-0a1e2ade0498\") " pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" Jul 10 22:22:56.593711 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.593563 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb67q\" (UniqueName: \"kubernetes.io/projected/f09d305a-458b-4609-8a90-0a1e2ade0498-kube-api-access-nb67q\") pod \"service-ca-79f68b9f4f-spd8d\" (UID: \"f09d305a-458b-4609-8a90-0a1e2ade0498\") " pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" Jul 10 22:22:56.593711 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.593634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f09d305a-458b-4609-8a90-0a1e2ade0498-signing-key\") pod \"service-ca-79f68b9f4f-spd8d\" (UID: \"f09d305a-458b-4609-8a90-0a1e2ade0498\") " pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" Jul 10 22:22:56.694337 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.694305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f09d305a-458b-4609-8a90-0a1e2ade0498-signing-key\") pod \"service-ca-79f68b9f4f-spd8d\" (UID: \"f09d305a-458b-4609-8a90-0a1e2ade0498\") " pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" Jul 10 22:22:56.694515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.694368 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f09d305a-458b-4609-8a90-0a1e2ade0498-signing-cabundle\") pod \"service-ca-79f68b9f4f-spd8d\" (UID: \"f09d305a-458b-4609-8a90-0a1e2ade0498\") " pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" Jul 10 22:22:56.694515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.694395 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb67q\" (UniqueName: \"kubernetes.io/projected/f09d305a-458b-4609-8a90-0a1e2ade0498-kube-api-access-nb67q\") pod \"service-ca-79f68b9f4f-spd8d\" (UID: \"f09d305a-458b-4609-8a90-0a1e2ade0498\") " pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" Jul 10 22:22:56.695011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.694992 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f09d305a-458b-4609-8a90-0a1e2ade0498-signing-cabundle\") pod \"service-ca-79f68b9f4f-spd8d\" (UID: \"f09d305a-458b-4609-8a90-0a1e2ade0498\") " pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" Jul 10 22:22:56.696608 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.696590 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f09d305a-458b-4609-8a90-0a1e2ade0498-signing-key\") pod \"service-ca-79f68b9f4f-spd8d\" (UID: \"f09d305a-458b-4609-8a90-0a1e2ade0498\") " pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" Jul 10 22:22:56.702452 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.702414 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb67q\" (UniqueName: \"kubernetes.io/projected/f09d305a-458b-4609-8a90-0a1e2ade0498-kube-api-access-nb67q\") pod \"service-ca-79f68b9f4f-spd8d\" (UID: \"f09d305a-458b-4609-8a90-0a1e2ade0498\") " pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" Jul 10 22:22:56.766096 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.766033 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" Jul 10 22:22:56.881725 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.881696 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-spd8d"] Jul 10 22:22:56.884239 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:22:56.884211 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09d305a_458b_4609_8a90_0a1e2ade0498.slice/crio-a4d5bd77103a53c7f7ac80fc53d49f645897c25106118533d71e0b23222b233c WatchSource:0}: Error finding container a4d5bd77103a53c7f7ac80fc53d49f645897c25106118533d71e0b23222b233c: Status 404 returned error can't find the container with id a4d5bd77103a53c7f7ac80fc53d49f645897c25106118533d71e0b23222b233c Jul 10 22:22:56.975294 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:56.975269 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7hj6f_1d1f9ea3-e113-4617-bd09-fc1b793a2bbe/node-ca/0.log" Jul 10 22:22:57.014272 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:57.014236 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" event={"ID":"f09d305a-458b-4609-8a90-0a1e2ade0498","Type":"ContainerStarted","Data":"1308969d64c33667809681f8a609b4c7671c8fc9a7a0407881db2d943ba153c9"} Jul 10 22:22:57.014412 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:57.014294 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" event={"ID":"f09d305a-458b-4609-8a90-0a1e2ade0498","Type":"ContainerStarted","Data":"a4d5bd77103a53c7f7ac80fc53d49f645897c25106118533d71e0b23222b233c"} Jul 10 22:22:57.038429 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:57.038333 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-spd8d" podStartSLOduration=1.038316159 podStartE2EDuration="1.038316159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:57.037103129 +0000 UTC m=+64.885385002" watchObservedRunningTime="2026-07-10 22:22:57.038316159 +0000 UTC m=+64.886598035" Jul 10 22:22:57.703334 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:57.703303 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:57.703697 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:57.703362 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:22:57.703697 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:57.703449 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 22:22:57.703697 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:57.703509 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm podName:fc1678f8-4523-4495-90d5-a35bd0fa292d nodeName:}" failed. No retries permitted until 2026-07-10 22:23:01.703494096 +0000 UTC m=+69.551775951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kcnnj" (UID: "fc1678f8-4523-4495-90d5-a35bd0fa292d") : configmap "kube-rbac-proxy" not found Jul 10 22:22:57.703697 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:57.703518 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 22:22:57.703697 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:57.703557 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls podName:fc1678f8-4523-4495-90d5-a35bd0fa292d nodeName:}" failed. No retries permitted until 2026-07-10 22:23:01.703546878 +0000 UTC m=+69.551828763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kcnnj" (UID: "fc1678f8-4523-4495-90d5-a35bd0fa292d") : secret "insights-runtime-extractor-tls" not found Jul 10 22:22:57.974368 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:57.974290 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s7t6h_8ad3159f-50bb-4fc7-af39-e85508a1575d/serve-healthcheck-canary/0.log" Jul 10 22:22:58.410326 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.410284 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:58.412967 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.412945 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 22:22:58.423699 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.423673 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/976133c2-7307-4130-a56d-56f9727c7ba6-original-pull-secret\") pod \"global-pull-secret-syncer-krhr5\" (UID: \"976133c2-7307-4130-a56d-56f9727c7ba6\") " pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:58.511575 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.511510 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbw4m\" (UniqueName: \"kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m\") pod \"network-check-target-98cjk\" (UID: \"43f8635d-832b-42d5-932b-5e06411e985a\") " pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:58.511756 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.511625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:22:58.514457 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.514221 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 22:22:58.515223 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.515201 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbw4m\" (UniqueName: \"kubernetes.io/projected/43f8635d-832b-42d5-932b-5e06411e985a-kube-api-access-nbw4m\") pod \"network-check-target-98cjk\" (UID: \"43f8635d-832b-42d5-932b-5e06411e985a\") " pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:58.522431 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:58.522407 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 22:22:58.522560 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:22:58.522500 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs podName:81ca9b9c-e95f-4ac1-9e97-96169222847b nodeName:}" failed. No retries permitted until 2026-07-10 22:24:02.522478718 +0000 UTC m=+130.370760578 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs") pod "network-metrics-daemon-wbxsz" (UID: "81ca9b9c-e95f-4ac1-9e97-96169222847b") : secret "metrics-daemon-secret" not found Jul 10 22:22:58.530826 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.530809 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-krhr5" Jul 10 22:22:58.537728 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.537708 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ts5wt\"" Jul 10 22:22:58.549774 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.549604 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:58.699240 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.699123 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-krhr5"] Jul 10 22:22:58.706545 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:22:58.706516 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod976133c2_7307_4130_a56d_56f9727c7ba6.slice/crio-4a5409227f24425c26e32ce4c91a47e10bae2cd3a6d1cf244a9eea97fc713fa0 WatchSource:0}: Error finding container 4a5409227f24425c26e32ce4c91a47e10bae2cd3a6d1cf244a9eea97fc713fa0: Status 404 returned error can't find the container with id 4a5409227f24425c26e32ce4c91a47e10bae2cd3a6d1cf244a9eea97fc713fa0 Jul 10 22:22:58.715282 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:58.715255 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-98cjk"] Jul 10 22:22:58.719321 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:22:58.719285 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f8635d_832b_42d5_932b_5e06411e985a.slice/crio-830a6f9889ecdc1c0baa885da218ab5a182e295023ae3776a73cab9ed66762ea WatchSource:0}: Error finding container 830a6f9889ecdc1c0baa885da218ab5a182e295023ae3776a73cab9ed66762ea: Status 404 returned error can't find the container with id 830a6f9889ecdc1c0baa885da218ab5a182e295023ae3776a73cab9ed66762ea Jul 10 22:22:59.023302 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:59.023214 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-98cjk" event={"ID":"43f8635d-832b-42d5-932b-5e06411e985a","Type":"ContainerStarted","Data":"d9f226ee0998dd91f53c7a4af1dd35fddded574cc63abb3a6c04dfca9a02c797"} Jul 10 22:22:59.023302 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:59.023269 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-98cjk" event={"ID":"43f8635d-832b-42d5-932b-5e06411e985a","Type":"ContainerStarted","Data":"830a6f9889ecdc1c0baa885da218ab5a182e295023ae3776a73cab9ed66762ea"} Jul 10 22:22:59.023559 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:59.023333 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:22:59.024707 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:59.024682 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-krhr5" event={"ID":"976133c2-7307-4130-a56d-56f9727c7ba6","Type":"ContainerStarted","Data":"4a5409227f24425c26e32ce4c91a47e10bae2cd3a6d1cf244a9eea97fc713fa0"} Jul 10 22:22:59.039509 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:22:59.039466 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-98cjk" podStartSLOduration=67.039432922 podStartE2EDuration="1m7.039432922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:22:59.037730047 +0000 UTC m=+66.886011928" watchObservedRunningTime="2026-07-10 22:22:59.039432922 +0000 UTC m=+66.887714800" Jul 10 22:23:00.328513 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:00.328479 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:23:00.328975 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:00.328595 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:23:00.328975 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:00.328661 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 10 22:23:00.328975 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:00.328685 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db6cd9c68-dft8m: secret "image-registry-tls" not found Jul 10 22:23:00.328975 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:00.328702 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 22:23:00.328975 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:00.328760 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls podName:a41d9dd7-0403-43e9-9ea9-11e18f8d32a4 nodeName:}" failed. No retries permitted until 2026-07-10 22:23:32.328736391 +0000 UTC m=+100.177018246 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls") pod "image-registry-7db6cd9c68-dft8m" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4") : secret "image-registry-tls" not found Jul 10 22:23:00.328975 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:00.328786 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls podName:4fc1d3dd-e770-4942-b849-c31dd37c52fa nodeName:}" failed. No retries permitted until 2026-07-10 22:23:32.328775514 +0000 UTC m=+100.177057367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls") pod "dns-default-8s7sv" (UID: "4fc1d3dd-e770-4942-b849-c31dd37c52fa") : secret "dns-default-metrics-tls" not found Jul 10 22:23:01.741389 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:01.741346 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:23:01.741870 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:01.741423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:23:01.741870 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:01.741496 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 22:23:01.741870 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:01.741546 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 22:23:01.741870 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:01.741589 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm podName:fc1678f8-4523-4495-90d5-a35bd0fa292d nodeName:}" failed. No retries permitted until 2026-07-10 22:23:09.741568481 +0000 UTC m=+77.589850342 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kcnnj" (UID: "fc1678f8-4523-4495-90d5-a35bd0fa292d") : configmap "kube-rbac-proxy" not found Jul 10 22:23:01.741870 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:01.741633 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls podName:fc1678f8-4523-4495-90d5-a35bd0fa292d nodeName:}" failed. No retries permitted until 2026-07-10 22:23:09.741623647 +0000 UTC m=+77.589905500 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kcnnj" (UID: "fc1678f8-4523-4495-90d5-a35bd0fa292d") : secret "insights-runtime-extractor-tls" not found Jul 10 22:23:03.163429 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:03.163381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cpfzk\" (UID: \"1604ab06-423e-4153-8138-6087e8f42f9a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:23:03.165799 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:03.165774 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/1604ab06-423e-4153-8138-6087e8f42f9a-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-cpfzk\" (UID: \"1604ab06-423e-4153-8138-6087e8f42f9a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:23:03.231245 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:03.231214 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-md7xf\"" Jul 10 22:23:03.239087 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:03.239065 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" Jul 10 22:23:03.374487 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:03.374300 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk"] Jul 10 22:23:04.044167 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:04.044135 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" event={"ID":"1604ab06-423e-4153-8138-6087e8f42f9a","Type":"ContainerStarted","Data":"6726fb2309b82a62f4470c79bf5e431170fdc078962e227bbaba4b19d5187d39"} Jul 10 22:23:04.045690 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:04.045662 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-krhr5" event={"ID":"976133c2-7307-4130-a56d-56f9727c7ba6","Type":"ContainerStarted","Data":"6cadf7b416187344002483463ed06824a6e0499eb4ad0df26a31a4f3d8379f3e"} Jul 10 22:23:04.059573 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:04.059525 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-krhr5" podStartSLOduration=66.517639353 podStartE2EDuration="1m11.059506042s" totalImagesPullingTime="4.541866689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:04.059485666 +0000 UTC m=+71.907767546" watchObservedRunningTime="2026-07-10 22:23:04.059506042 +0000 UTC m=+71.907787919" Jul 10 22:23:06.056466 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:06.056420 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" event={"ID":"1604ab06-423e-4153-8138-6087e8f42f9a","Type":"ContainerStarted","Data":"91e981fadd6a58bca2cb779cf3dc55567b6133de12cbfe5a7ffa68be3f12f879"} Jul 10 22:23:06.056823 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:06.056483 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" event={"ID":"1604ab06-423e-4153-8138-6087e8f42f9a","Type":"ContainerStarted","Data":"855278a8f99d87062d34b50d92f59111b40e61df8692357418b1f1448725c124"} Jul 10 22:23:06.072011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:06.071973 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-cpfzk" podStartSLOduration=17.13996144 podStartE2EDuration="19.071958939s" totalImagesPullingTime="1.931997499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:06.07165866 +0000 UTC m=+73.919940547" watchObservedRunningTime="2026-07-10 22:23:06.071958939 +0000 UTC m=+73.920240813" Jul 10 22:23:09.819678 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:09.819636 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:23:09.820156 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:09.819708 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:23:09.820156 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:09.820139 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fc1678f8-4523-4495-90d5-a35bd0fa292d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:23:09.822010 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:09.821987 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fc1678f8-4523-4495-90d5-a35bd0fa292d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kcnnj\" (UID: \"fc1678f8-4523-4495-90d5-a35bd0fa292d\") " pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:23:09.904198 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:09.904173 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kcnnj" Jul 10 22:23:10.025327 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:10.025306 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kcnnj"] Jul 10 22:23:10.028072 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:10.028042 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1678f8_4523_4495_90d5_a35bd0fa292d.slice/crio-eecb7f462bac3134efba3dc32672fb95e9ef3a943a8b274342c667536a59a159 WatchSource:0}: Error finding container eecb7f462bac3134efba3dc32672fb95e9ef3a943a8b274342c667536a59a159: Status 404 returned error can't find the container with id eecb7f462bac3134efba3dc32672fb95e9ef3a943a8b274342c667536a59a159 Jul 10 22:23:10.071200 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:10.071111 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kcnnj" event={"ID":"fc1678f8-4523-4495-90d5-a35bd0fa292d","Type":"ContainerStarted","Data":"eecb7f462bac3134efba3dc32672fb95e9ef3a943a8b274342c667536a59a159"} Jul 10 22:23:14.086925 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:14.086837 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kcnnj" event={"ID":"fc1678f8-4523-4495-90d5-a35bd0fa292d","Type":"ContainerStarted","Data":"37a864a3ed23bedb7435ff1368bacb810a3e8b2cd41f4b952634c7c57c3111f7"} Jul 10 22:23:15.092565 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:15.092455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kcnnj" event={"ID":"fc1678f8-4523-4495-90d5-a35bd0fa292d","Type":"ContainerStarted","Data":"d600cb00643acf803b087056e27bf964cfa12f8ae758913e516595dd9e831bf8"} Jul 10 22:23:16.463483 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.462701 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69"] Jul 10 22:23:16.475805 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.475781 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:16.480576 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.480482 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 22:23:16.481767 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.481745 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 22:23:16.482103 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.482076 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 22:23:16.482833 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.482665 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 22:23:16.493062 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.493039 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69"] Jul 10 22:23:16.572154 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.572118 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db09ae19-3662-40de-af6c-f375aead9bfc-tmp\") pod \"klusterlet-addon-workmgr-7f75b7f5f7-t4w69\" (UID: \"db09ae19-3662-40de-af6c-f375aead9bfc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:16.572323 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.572158 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/db09ae19-3662-40de-af6c-f375aead9bfc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f75b7f5f7-t4w69\" (UID: \"db09ae19-3662-40de-af6c-f375aead9bfc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:16.572323 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.572188 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txz8d\" (UniqueName: \"kubernetes.io/projected/db09ae19-3662-40de-af6c-f375aead9bfc-kube-api-access-txz8d\") pod \"klusterlet-addon-workmgr-7f75b7f5f7-t4w69\" (UID: \"db09ae19-3662-40de-af6c-f375aead9bfc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:16.672865 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.672829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/db09ae19-3662-40de-af6c-f375aead9bfc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f75b7f5f7-t4w69\" (UID: \"db09ae19-3662-40de-af6c-f375aead9bfc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:16.672865 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.672864 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txz8d\" (UniqueName: \"kubernetes.io/projected/db09ae19-3662-40de-af6c-f375aead9bfc-kube-api-access-txz8d\") pod \"klusterlet-addon-workmgr-7f75b7f5f7-t4w69\" (UID: \"db09ae19-3662-40de-af6c-f375aead9bfc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:16.673098 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.673075 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db09ae19-3662-40de-af6c-f375aead9bfc-tmp\") pod \"klusterlet-addon-workmgr-7f75b7f5f7-t4w69\" (UID: \"db09ae19-3662-40de-af6c-f375aead9bfc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:16.673390 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.673369 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db09ae19-3662-40de-af6c-f375aead9bfc-tmp\") pod \"klusterlet-addon-workmgr-7f75b7f5f7-t4w69\" (UID: \"db09ae19-3662-40de-af6c-f375aead9bfc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:16.675241 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.675213 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/db09ae19-3662-40de-af6c-f375aead9bfc-klusterlet-config\") pod \"klusterlet-addon-workmgr-7f75b7f5f7-t4w69\" (UID: \"db09ae19-3662-40de-af6c-f375aead9bfc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:16.684548 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.684527 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txz8d\" (UniqueName: \"kubernetes.io/projected/db09ae19-3662-40de-af6c-f375aead9bfc-kube-api-access-txz8d\") pod \"klusterlet-addon-workmgr-7f75b7f5f7-t4w69\" (UID: \"db09ae19-3662-40de-af6c-f375aead9bfc\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:16.786389 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.786369 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:16.920645 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:16.920611 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69"] Jul 10 22:23:16.923507 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:16.923482 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb09ae19_3662_40de_af6c_f375aead9bfc.slice/crio-bea224bf6554133223de2fdc75ad1d9a5695700cff7dc7ea46a660d08dbeb4dd WatchSource:0}: Error finding container bea224bf6554133223de2fdc75ad1d9a5695700cff7dc7ea46a660d08dbeb4dd: Status 404 returned error can't find the container with id bea224bf6554133223de2fdc75ad1d9a5695700cff7dc7ea46a660d08dbeb4dd Jul 10 22:23:17.100830 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:17.100794 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kcnnj" event={"ID":"fc1678f8-4523-4495-90d5-a35bd0fa292d","Type":"ContainerStarted","Data":"ab7fac72846001c27ddf9cd2cf913966d60cf19a1b8e4e282e831de2f4b22c5d"} Jul 10 22:23:17.101884 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:17.101866 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" event={"ID":"db09ae19-3662-40de-af6c-f375aead9bfc","Type":"ContainerStarted","Data":"bea224bf6554133223de2fdc75ad1d9a5695700cff7dc7ea46a660d08dbeb4dd"} Jul 10 22:23:17.122893 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:17.122846 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kcnnj" podStartSLOduration=17.539136273 podStartE2EDuration="24.122833086s" totalImagesPullingTime="6.583696813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 22:22:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:17.122067305 +0000 UTC m=+84.970349182" watchObservedRunningTime="2026-07-10 22:23:17.122833086 +0000 UTC m=+84.971114961" Jul 10 22:23:21.120384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:21.120344 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" event={"ID":"db09ae19-3662-40de-af6c-f375aead9bfc","Type":"ContainerStarted","Data":"d9683c46c16c5d0cba64d835e807e30e3c968ad2371da8f2858fbf9afc31f2fa"} Jul 10 22:23:21.120818 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:21.120555 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:21.122249 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:21.122227 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" Jul 10 22:23:21.136151 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:21.136110 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7f75b7f5f7-t4w69" podStartSLOduration=1.6006033469999998 podStartE2EDuration="5.136097293s" totalImagesPullingTime="3.535493946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:21.135273817 +0000 UTC m=+88.983555706" watchObservedRunningTime="2026-07-10 22:23:21.136097293 +0000 UTC m=+88.984379166" Jul 10 22:23:23.318813 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:23.318781 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62"] Jul 10 22:23:23.322010 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:23.321994 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62" Jul 10 22:23:23.324361 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:23.324337 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 10 22:23:23.324623 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:23.324608 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ssbss\"" Jul 10 22:23:23.331358 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:23.331332 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62"] Jul 10 22:23:23.429167 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:23.429134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef7cec55-1888-4b4d-bc27-c45e37991d6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-g7v62\" (UID: \"ef7cec55-1888-4b4d-bc27-c45e37991d6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62" Jul 10 22:23:23.529843 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:23.529816 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef7cec55-1888-4b4d-bc27-c45e37991d6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-g7v62\" (UID: \"ef7cec55-1888-4b4d-bc27-c45e37991d6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62" Jul 10 22:23:23.532373 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:23.532349 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/ef7cec55-1888-4b4d-bc27-c45e37991d6e-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-g7v62\" (UID: \"ef7cec55-1888-4b4d-bc27-c45e37991d6e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62" Jul 10 22:23:23.630929 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:23.630904 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62" Jul 10 22:23:23.749318 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:23.749287 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62"] Jul 10 22:23:23.752829 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:23.752791 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7cec55_1888_4b4d_bc27_c45e37991d6e.slice/crio-7b5d3c26b3fb1b5ff18a8f910cafeacf328b3a0106a079465b3d57e78719ed6c WatchSource:0}: Error finding container 7b5d3c26b3fb1b5ff18a8f910cafeacf328b3a0106a079465b3d57e78719ed6c: Status 404 returned error can't find the container with id 7b5d3c26b3fb1b5ff18a8f910cafeacf328b3a0106a079465b3d57e78719ed6c Jul 10 22:23:24.132330 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:24.132297 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62" event={"ID":"ef7cec55-1888-4b4d-bc27-c45e37991d6e","Type":"ContainerStarted","Data":"7b5d3c26b3fb1b5ff18a8f910cafeacf328b3a0106a079465b3d57e78719ed6c"} Jul 10 22:23:26.140155 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:26.140118 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62" event={"ID":"ef7cec55-1888-4b4d-bc27-c45e37991d6e","Type":"ContainerStarted","Data":"447479bec3e51c1aa098a0a39aa1d4c05669fa96aba2b07114895dc68e66f467"} Jul 10 22:23:26.140667 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:26.140282 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62" Jul 10 22:23:26.144936 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:26.144912 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62" Jul 10 22:23:26.155561 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:26.155518 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-g7v62" podStartSLOduration=1.368435987 podStartE2EDuration="3.155501191s" totalImagesPullingTime="1.787065204s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:26.1549939 +0000 UTC m=+94.003275777" watchObservedRunningTime="2026-07-10 22:23:26.155501191 +0000 UTC m=+94.003783066" Jul 10 22:23:30.031566 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.031537 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-98cjk" Jul 10 22:23:30.761030 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.760999 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-97znx"] Jul 10 22:23:30.763346 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.763324 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.765610 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.765586 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 10 22:23:30.765773 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.765759 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8xqrm\"" Jul 10 22:23:30.765838 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.765608 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 10 22:23:30.765917 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.765693 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 22:23:30.766518 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.766499 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 22:23:30.766609 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.766510 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 22:23:30.777743 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.777718 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-97znx"] Jul 10 22:23:30.800416 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.800395 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vbwlt"] Jul 10 22:23:30.802468 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.802452 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.805511 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.804993 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 22:23:30.805511 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.805056 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 22:23:30.805511 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.805402 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 22:23:30.805511 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.805432 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jcm6v\"" Jul 10 22:23:30.884580 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884546 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-textfile\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.884755 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07c406dc-3ad9-4c9b-964d-639136a35225-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.884755 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884628 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07c406dc-3ad9-4c9b-964d-639136a35225-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.884755 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884714 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7wkd\" (UniqueName: \"kubernetes.io/projected/07c406dc-3ad9-4c9b-964d-639136a35225-kube-api-access-w7wkd\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.884912 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-tls\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.884912 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884785 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02307dd9-985a-402a-af0d-be3ed719cda6-metrics-client-ca\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.884912 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884809 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-wtmp\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.884912 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.884912 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884894 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.885142 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884920 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbjsv\" (UniqueName: \"kubernetes.io/projected/02307dd9-985a-402a-af0d-be3ed719cda6-kube-api-access-tbjsv\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.885142 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884968 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07c406dc-3ad9-4c9b-964d-639136a35225-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.885142 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.884995 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02307dd9-985a-402a-af0d-be3ed719cda6-sys\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.885142 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.885015 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02307dd9-985a-402a-af0d-be3ed719cda6-root\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985623 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7wkd\" (UniqueName: \"kubernetes.io/projected/07c406dc-3ad9-4c9b-964d-639136a35225-kube-api-access-w7wkd\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-tls\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02307dd9-985a-402a-af0d-be3ed719cda6-metrics-client-ca\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985733 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-wtmp\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985757 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985784 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985810 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbjsv\" (UniqueName: \"kubernetes.io/projected/02307dd9-985a-402a-af0d-be3ed719cda6-kube-api-access-tbjsv\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07c406dc-3ad9-4c9b-964d-639136a35225-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985885 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02307dd9-985a-402a-af0d-be3ed719cda6-sys\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985906 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02307dd9-985a-402a-af0d-be3ed719cda6-root\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985964 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-textfile\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.985994 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07c406dc-3ad9-4c9b-964d-639136a35225-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.986555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.986028 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07c406dc-3ad9-4c9b-964d-639136a35225-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.988511 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.987971 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-wtmp\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.988511 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:30.988198 2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 22:23:30.988511 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.988217 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/07c406dc-3ad9-4c9b-964d-639136a35225-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.988511 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:30.988275 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-tls podName:02307dd9-985a-402a-af0d-be3ed719cda6 nodeName:}" failed. No retries permitted until 2026-07-10 22:23:31.488255555 +0000 UTC m=+99.336537422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-tls") pod "node-exporter-vbwlt" (UID: "02307dd9-985a-402a-af0d-be3ed719cda6") : secret "node-exporter-tls" not found Jul 10 22:23:30.988511 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.988352 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02307dd9-985a-402a-af0d-be3ed719cda6-metrics-client-ca\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.988511 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.988420 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/02307dd9-985a-402a-af0d-be3ed719cda6-sys\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.988511 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.988470 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-accelerators-collector-config\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.988910 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.988632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-textfile\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.988996 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.988979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/02307dd9-985a-402a-af0d-be3ed719cda6-root\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.992710 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.992685 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:30.992994 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.992970 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/07c406dc-3ad9-4c9b-964d-639136a35225-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.993089 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.993043 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/07c406dc-3ad9-4c9b-964d-639136a35225-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.995855 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.995829 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7wkd\" (UniqueName: \"kubernetes.io/projected/07c406dc-3ad9-4c9b-964d-639136a35225-kube-api-access-w7wkd\") pod \"openshift-state-metrics-84bd68b896-97znx\" (UID: \"07c406dc-3ad9-4c9b-964d-639136a35225\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:30.997089 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:30.997067 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbjsv\" (UniqueName: \"kubernetes.io/projected/02307dd9-985a-402a-af0d-be3ed719cda6-kube-api-access-tbjsv\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:31.075326 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:31.075243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" Jul 10 22:23:31.212119 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:31.212090 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-97znx"] Jul 10 22:23:31.215085 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:31.215057 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07c406dc_3ad9_4c9b_964d_639136a35225.slice/crio-c1e372c6947cc2a31a2691eb42d7f874934e6e751704d454d319bb03ae3da02c WatchSource:0}: Error finding container c1e372c6947cc2a31a2691eb42d7f874934e6e751704d454d319bb03ae3da02c: Status 404 returned error can't find the container with id c1e372c6947cc2a31a2691eb42d7f874934e6e751704d454d319bb03ae3da02c Jul 10 22:23:31.491314 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:31.491275 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-tls\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:31.493536 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:31.493517 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/02307dd9-985a-402a-af0d-be3ed719cda6-node-exporter-tls\") pod \"node-exporter-vbwlt\" (UID: \"02307dd9-985a-402a-af0d-be3ed719cda6\") " pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:31.712864 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:31.712834 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vbwlt" Jul 10 22:23:31.721738 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:31.721713 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02307dd9_985a_402a_af0d_be3ed719cda6.slice/crio-80f9f7a9d8b927d06751d29f307558e4508111cfab6a19273caffa9b0e675bd2 WatchSource:0}: Error finding container 80f9f7a9d8b927d06751d29f307558e4508111cfab6a19273caffa9b0e675bd2: Status 404 returned error can't find the container with id 80f9f7a9d8b927d06751d29f307558e4508111cfab6a19273caffa9b0e675bd2 Jul 10 22:23:32.162698 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.162649 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" event={"ID":"07c406dc-3ad9-4c9b-964d-639136a35225","Type":"ContainerStarted","Data":"b3d940981276c30f6bf5d4f2f72ea2c1a2999f5d4c73e2fe244369021efae33e"} Jul 10 22:23:32.163144 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.162753 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" event={"ID":"07c406dc-3ad9-4c9b-964d-639136a35225","Type":"ContainerStarted","Data":"e45a807ac7a925ae8323f4aa53286a3845cf553db6c6c7112bf262fa34f19673"} Jul 10 22:23:32.163144 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.162792 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" event={"ID":"07c406dc-3ad9-4c9b-964d-639136a35225","Type":"ContainerStarted","Data":"c1e372c6947cc2a31a2691eb42d7f874934e6e751704d454d319bb03ae3da02c"} Jul 10 22:23:32.167484 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.167411 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbwlt" event={"ID":"02307dd9-985a-402a-af0d-be3ed719cda6","Type":"ContainerStarted","Data":"80f9f7a9d8b927d06751d29f307558e4508111cfab6a19273caffa9b0e675bd2"} Jul 10 22:23:32.400587 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.400546 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:23:32.400766 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.400697 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:23:32.403779 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.403754 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4fc1d3dd-e770-4942-b849-c31dd37c52fa-metrics-tls\") pod \"dns-default-8s7sv\" (UID: \"4fc1d3dd-e770-4942-b849-c31dd37c52fa\") " pod="openshift-dns/dns-default-8s7sv" Jul 10 22:23:32.404338 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.404291 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") pod \"image-registry-7db6cd9c68-dft8m\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:23:32.426510 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.426408 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5zdmx\"" Jul 10 22:23:32.436147 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.436111 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:23:32.442378 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.442356 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-t9ksf\"" Jul 10 22:23:32.450285 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.450262 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8s7sv" Jul 10 22:23:32.729100 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.729077 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86b8dff754-cxzkd"] Jul 10 22:23:32.733356 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.733210 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.736544 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.736109 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 10 22:23:32.736544 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.736320 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 10 22:23:32.737224 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.736871 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5vvo8dm9p8ep2\"" Jul 10 22:23:32.737224 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.736994 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-bbg47\"" Jul 10 22:23:32.737224 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.737074 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 10 22:23:32.737224 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.736997 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 10 22:23:32.737224 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.737084 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 10 22:23:32.742428 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.742349 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86b8dff754-cxzkd"] Jul 10 22:23:32.804035 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.804003 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.804177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.804070 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.804177 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.804105 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.804264 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.804195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.804264 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.804249 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkvdh\" (UniqueName: \"kubernetes.io/projected/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-kube-api-access-rkvdh\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.804345 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.804326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-tls\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.804381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.804360 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-metrics-client-ca\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.804424 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.804386 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-grpc-tls\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.810099 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.810077 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7db6cd9c68-dft8m"] Jul 10 22:23:32.812515 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:32.812488 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda41d9dd7_0403_43e9_9ea9_11e18f8d32a4.slice/crio-604f13d759da6c1dd32238cbb3e5060c9de5ccaf8a79aef9f7446a3e62ed9f47 WatchSource:0}: Error finding container 604f13d759da6c1dd32238cbb3e5060c9de5ccaf8a79aef9f7446a3e62ed9f47: Status 404 returned error can't find the container with id 604f13d759da6c1dd32238cbb3e5060c9de5ccaf8a79aef9f7446a3e62ed9f47 Jul 10 22:23:32.824518 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.824497 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8s7sv"] Jul 10 22:23:32.829398 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:32.829367 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc1d3dd_e770_4942_b849_c31dd37c52fa.slice/crio-edfbd62c3b04e6e1067c61bbf171961d1ad42cb7ac5786711ff38298f9f5fe3c WatchSource:0}: Error finding container edfbd62c3b04e6e1067c61bbf171961d1ad42cb7ac5786711ff38298f9f5fe3c: Status 404 returned error can't find the container with id edfbd62c3b04e6e1067c61bbf171961d1ad42cb7ac5786711ff38298f9f5fe3c Jul 10 22:23:32.904861 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.904834 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkvdh\" (UniqueName: \"kubernetes.io/projected/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-kube-api-access-rkvdh\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.905109 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.904887 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-tls\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.905109 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.904920 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-metrics-client-ca\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.905109 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.904950 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-grpc-tls\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.905109 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.905005 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.905109 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.905050 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.905109 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.905089 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.905505 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.905141 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.905762 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.905734 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-metrics-client-ca\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.907993 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.907944 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.908095 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.908068 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.908170 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.908104 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-tls\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.908416 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.908394 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.908541 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.908525 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-grpc-tls\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.908756 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.908740 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:32.914189 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:32.914156 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkvdh\" (UniqueName: \"kubernetes.io/projected/0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff-kube-api-access-rkvdh\") pod \"thanos-querier-86b8dff754-cxzkd\" (UID: \"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff\") " pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:33.048823 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:33.048741 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:33.172183 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:33.172147 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8s7sv" event={"ID":"4fc1d3dd-e770-4942-b849-c31dd37c52fa","Type":"ContainerStarted","Data":"edfbd62c3b04e6e1067c61bbf171961d1ad42cb7ac5786711ff38298f9f5fe3c"} Jul 10 22:23:33.174471 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:33.174424 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" event={"ID":"07c406dc-3ad9-4c9b-964d-639136a35225","Type":"ContainerStarted","Data":"2624d460716f509ef3c75e39716ce3a0503ef3dd329331af0aa644d9c685858c"} Jul 10 22:23:33.176040 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:33.176010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" event={"ID":"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4","Type":"ContainerStarted","Data":"5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d"} Jul 10 22:23:33.176163 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:33.176068 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:23:33.176163 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:33.176090 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" event={"ID":"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4","Type":"ContainerStarted","Data":"604f13d759da6c1dd32238cbb3e5060c9de5ccaf8a79aef9f7446a3e62ed9f47"} Jul 10 22:23:33.177506 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:33.177482 2559 generic.go:356] "Generic (PLEG): container finished" podID="02307dd9-985a-402a-af0d-be3ed719cda6" containerID="13c73b14a24286888438ae1826cfc5c8d4b5a6303dfd115b3cdfa8e1478b9a84" exitCode=0 Jul 10 22:23:33.177629 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:33.177515 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbwlt" event={"ID":"02307dd9-985a-402a-af0d-be3ed719cda6","Type":"ContainerDied","Data":"13c73b14a24286888438ae1826cfc5c8d4b5a6303dfd115b3cdfa8e1478b9a84"} Jul 10 22:23:33.183849 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:33.183829 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86b8dff754-cxzkd"] Jul 10 22:23:33.187033 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:33.187011 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0476ccfb_d276_4cf0_bd9f_32fdc2ebc7ff.slice/crio-b7bdbed491a9ed91363991789fa14b9982de15448c0e83313a914719bcf8c4bd WatchSource:0}: Error finding container b7bdbed491a9ed91363991789fa14b9982de15448c0e83313a914719bcf8c4bd: Status 404 returned error can't find the container with id b7bdbed491a9ed91363991789fa14b9982de15448c0e83313a914719bcf8c4bd Jul 10 22:23:33.197646 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:33.197604 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-97znx" podStartSLOduration=1.8523982700000001 podStartE2EDuration="3.197590998s" totalImagesPullingTime="1.345192728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:33.197365918 +0000 UTC m=+101.045647806" watchObservedRunningTime="2026-07-10 22:23:33.197590998 +0000 UTC m=+101.045872871" Jul 10 22:23:33.221020 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:33.220969 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" podStartSLOduration=100.220947522 podStartE2EDuration="1m40.220947522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:33.218817906 +0000 UTC m=+101.067099783" watchObservedRunningTime="2026-07-10 22:23:33.220947522 +0000 UTC m=+101.069229398" Jul 10 22:23:34.184979 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:34.184932 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbwlt" event={"ID":"02307dd9-985a-402a-af0d-be3ed719cda6","Type":"ContainerStarted","Data":"8eb0399a32f13b7f08e6a6693b555c3c97e7559fe9d53794a2142935291408df"} Jul 10 22:23:34.185460 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:34.184996 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vbwlt" event={"ID":"02307dd9-985a-402a-af0d-be3ed719cda6","Type":"ContainerStarted","Data":"399fa2eec49bba9658b8f70bdc2306d4fb2f8eb36bfaf7e85e067b573a02415b"} Jul 10 22:23:34.186630 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:34.186593 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" event={"ID":"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff","Type":"ContainerStarted","Data":"b7bdbed491a9ed91363991789fa14b9982de15448c0e83313a914719bcf8c4bd"} Jul 10 22:23:34.208381 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:34.208326 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vbwlt" podStartSLOduration=3.188495855 podStartE2EDuration="4.208308975s" totalImagesPullingTime="939.355105ms" totalInitContainerRuntime="80.458015ms" isStatefulPod=true podCreationTimestamp="2026-07-10 22:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:34.207356729 +0000 UTC m=+102.055638605" watchObservedRunningTime="2026-07-10 22:23:34.208308975 +0000 UTC m=+102.056590894" Jul 10 22:23:35.104414 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.104367 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6d4999c9b9-mb5wx"] Jul 10 22:23:35.108029 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.107999 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.110581 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.110546 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 10 22:23:35.110737 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.110605 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 10 22:23:35.110737 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.110642 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 10 22:23:35.110929 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.110911 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 10 22:23:35.111805 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.111788 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6l6tt\"" Jul 10 22:23:35.111927 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.111915 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-af4q04no9n620\"" Jul 10 22:23:35.117175 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.117151 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d4999c9b9-mb5wx"] Jul 10 22:23:35.228684 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.228653 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-secret-metrics-server-tls\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.229091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.228693 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-metrics-server-audit-profiles\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.229091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.228808 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6lf2\" (UniqueName: \"kubernetes.io/projected/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-kube-api-access-j6lf2\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.229091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.228862 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.229091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.228960 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-client-ca-bundle\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.229091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.229032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-secret-metrics-server-client-certs\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.229091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.229065 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-audit-log\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.329537 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.329501 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6lf2\" (UniqueName: \"kubernetes.io/projected/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-kube-api-access-j6lf2\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.329713 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.329557 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.329713 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.329595 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-client-ca-bundle\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.329713 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.329647 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-secret-metrics-server-client-certs\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.329868 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.329779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-audit-log\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.329868 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.329840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-secret-metrics-server-tls\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.329957 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.329876 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-metrics-server-audit-profiles\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.330466 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.330249 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-audit-log\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.330725 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.330660 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.331131 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.331106 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-metrics-server-audit-profiles\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.332962 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.332754 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-secret-metrics-server-client-certs\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.332962 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.332925 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-client-ca-bundle\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.333108 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.333051 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-secret-metrics-server-tls\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.336962 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.336940 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6lf2\" (UniqueName: \"kubernetes.io/projected/c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec-kube-api-access-j6lf2\") pod \"metrics-server-6d4999c9b9-mb5wx\" (UID: \"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec\") " pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.419980 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.419892 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:35.535063 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.535028 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl"] Jul 10 22:23:35.549036 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.549007 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl"] Jul 10 22:23:35.549192 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.549142 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" Jul 10 22:23:35.551531 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.551299 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 10 22:23:35.551854 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.551829 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qfdd7\"" Jul 10 22:23:35.633376 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.633342 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d8d7197-6c1f-4feb-a515-fded7e132782-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-vn2nl\" (UID: \"1d8d7197-6c1f-4feb-a515-fded7e132782\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" Jul 10 22:23:35.633565 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.633382 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w756k\" (UniqueName: \"kubernetes.io/projected/1d8d7197-6c1f-4feb-a515-fded7e132782-kube-api-access-w756k\") pod \"monitoring-plugin-5f68df7f58-vn2nl\" (UID: \"1d8d7197-6c1f-4feb-a515-fded7e132782\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" Jul 10 22:23:35.734352 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.734314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d8d7197-6c1f-4feb-a515-fded7e132782-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-vn2nl\" (UID: \"1d8d7197-6c1f-4feb-a515-fded7e132782\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" Jul 10 22:23:35.734605 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.734367 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w756k\" (UniqueName: \"kubernetes.io/projected/1d8d7197-6c1f-4feb-a515-fded7e132782-kube-api-access-w756k\") pod \"monitoring-plugin-5f68df7f58-vn2nl\" (UID: \"1d8d7197-6c1f-4feb-a515-fded7e132782\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" Jul 10 22:23:35.734873 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:35.734749 2559 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 10 22:23:35.734873 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:23:35.734848 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d8d7197-6c1f-4feb-a515-fded7e132782-monitoring-plugin-cert podName:1d8d7197-6c1f-4feb-a515-fded7e132782 nodeName:}" failed. No retries permitted until 2026-07-10 22:23:36.234804279 +0000 UTC m=+104.083086136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/1d8d7197-6c1f-4feb-a515-fded7e132782-monitoring-plugin-cert") pod "monitoring-plugin-5f68df7f58-vn2nl" (UID: "1d8d7197-6c1f-4feb-a515-fded7e132782") : secret "monitoring-plugin-cert" not found Jul 10 22:23:35.743386 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.743367 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w756k\" (UniqueName: \"kubernetes.io/projected/1d8d7197-6c1f-4feb-a515-fded7e132782-kube-api-access-w756k\") pod \"monitoring-plugin-5f68df7f58-vn2nl\" (UID: \"1d8d7197-6c1f-4feb-a515-fded7e132782\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" Jul 10 22:23:35.837006 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.836907 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d4999c9b9-mb5wx"] Jul 10 22:23:35.838905 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:35.838879 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4edbd6a_9a1b_4bf3_8653_aaeaecc00dec.slice/crio-efc8b94dbdcd69ea50c5dfedb64dbbb8910947fae5884dacfdbbea6f85946d17 WatchSource:0}: Error finding container efc8b94dbdcd69ea50c5dfedb64dbbb8910947fae5884dacfdbbea6f85946d17: Status 404 returned error can't find the container with id efc8b94dbdcd69ea50c5dfedb64dbbb8910947fae5884dacfdbbea6f85946d17 Jul 10 22:23:35.983732 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.983702 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-f44675f6-c7lbg"] Jul 10 22:23:35.986452 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.986415 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:35.989810 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.989080 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 10 22:23:35.989810 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.989691 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 10 22:23:35.989977 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.989828 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-jnt5m\"" Jul 10 22:23:35.989977 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.989892 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 10 22:23:35.990077 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.989994 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 10 22:23:35.990319 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.990301 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 10 22:23:35.995831 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:35.995766 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 10 22:23:36.001512 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.001491 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f44675f6-c7lbg"] Jul 10 22:23:36.037218 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.037196 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-telemeter-client-tls\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.037320 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.037237 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-secret-telemeter-client\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.037320 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.037275 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-federate-client-tls\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.037429 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.037326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0290b83d-bf15-4cc2-aac0-7daeaaf40227-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.037429 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.037414 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0290b83d-bf15-4cc2-aac0-7daeaaf40227-metrics-client-ca\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.037544 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.037453 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59lwq\" (UniqueName: \"kubernetes.io/projected/0290b83d-bf15-4cc2-aac0-7daeaaf40227-kube-api-access-59lwq\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.037544 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.037510 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0290b83d-bf15-4cc2-aac0-7daeaaf40227-serving-certs-ca-bundle\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.037544 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.037537 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.138979 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.138932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0290b83d-bf15-4cc2-aac0-7daeaaf40227-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.139158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.139016 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0290b83d-bf15-4cc2-aac0-7daeaaf40227-metrics-client-ca\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.139158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.139036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59lwq\" (UniqueName: \"kubernetes.io/projected/0290b83d-bf15-4cc2-aac0-7daeaaf40227-kube-api-access-59lwq\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.139158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.139072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0290b83d-bf15-4cc2-aac0-7daeaaf40227-serving-certs-ca-bundle\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.139158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.139088 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.139158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.139113 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-telemeter-client-tls\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.139158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.139143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-secret-telemeter-client\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.139529 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.139193 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-federate-client-tls\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.139985 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.139937 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0290b83d-bf15-4cc2-aac0-7daeaaf40227-metrics-client-ca\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.140121 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.140023 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0290b83d-bf15-4cc2-aac0-7daeaaf40227-serving-certs-ca-bundle\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.140121 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.140095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0290b83d-bf15-4cc2-aac0-7daeaaf40227-telemeter-trusted-ca-bundle\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.141648 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.141614 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-federate-client-tls\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.141763 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.141739 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-telemeter-client-tls\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.141877 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.141856 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.141877 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.141872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0290b83d-bf15-4cc2-aac0-7daeaaf40227-secret-telemeter-client\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.146669 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.146650 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59lwq\" (UniqueName: \"kubernetes.io/projected/0290b83d-bf15-4cc2-aac0-7daeaaf40227-kube-api-access-59lwq\") pod \"telemeter-client-f44675f6-c7lbg\" (UID: \"0290b83d-bf15-4cc2-aac0-7daeaaf40227\") " pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.196693 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.196663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8s7sv" event={"ID":"4fc1d3dd-e770-4942-b849-c31dd37c52fa","Type":"ContainerStarted","Data":"778762bc8480c2966174a60dfe0521ec54f7ab073c70c02e00f33ea5b3474902"} Jul 10 22:23:36.196851 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.196709 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8s7sv" event={"ID":"4fc1d3dd-e770-4942-b849-c31dd37c52fa","Type":"ContainerStarted","Data":"f06b7c4ac6f197eb6c4eb5e82c96c82d71e8a65d21ca6fe9e1a4d855d861ec7b"} Jul 10 22:23:36.196851 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.196769 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8s7sv" Jul 10 22:23:36.197883 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.197860 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" event={"ID":"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec","Type":"ContainerStarted","Data":"efc8b94dbdcd69ea50c5dfedb64dbbb8910947fae5884dacfdbbea6f85946d17"} Jul 10 22:23:36.199759 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.199737 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" event={"ID":"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff","Type":"ContainerStarted","Data":"37bafb04f37ce69d599fd1b7d0e2152b8ab86cd1df0af07838066ad3a1d797ec"} Jul 10 22:23:36.199842 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.199787 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" event={"ID":"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff","Type":"ContainerStarted","Data":"d40bffd08c9095da6565045631a11cf9d2fe0cf226cbe0f10c7e600961d84b8e"} Jul 10 22:23:36.199842 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.199812 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" event={"ID":"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff","Type":"ContainerStarted","Data":"02d50748c85cda10ac63c432c02796d6d54b43d257fb738c007e14256d25d119"} Jul 10 22:23:36.212945 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.212873 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8s7sv" podStartSLOduration=65.341873727 podStartE2EDuration="1m8.212862343s" totalImagesPullingTime="2.870988616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:22:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:36.212411167 +0000 UTC m=+104.060693052" watchObservedRunningTime="2026-07-10 22:23:36.212862343 +0000 UTC m=+104.061144196" Jul 10 22:23:36.240170 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.240132 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d8d7197-6c1f-4feb-a515-fded7e132782-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-vn2nl\" (UID: \"1d8d7197-6c1f-4feb-a515-fded7e132782\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" Jul 10 22:23:36.242385 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.242361 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1d8d7197-6c1f-4feb-a515-fded7e132782-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-vn2nl\" (UID: \"1d8d7197-6c1f-4feb-a515-fded7e132782\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" Jul 10 22:23:36.308125 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.308094 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" Jul 10 22:23:36.445075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.445041 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-f44675f6-c7lbg"] Jul 10 22:23:36.449012 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:36.448981 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0290b83d_bf15_4cc2_aac0_7daeaaf40227.slice/crio-f50544b715abcadae0803208ea70f7d8f509d8cfbe2fb97ef6aad6edca282d8b WatchSource:0}: Error finding container f50544b715abcadae0803208ea70f7d8f509d8cfbe2fb97ef6aad6edca282d8b: Status 404 returned error can't find the container with id f50544b715abcadae0803208ea70f7d8f509d8cfbe2fb97ef6aad6edca282d8b Jul 10 22:23:36.461622 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.461600 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" Jul 10 22:23:36.587902 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.587878 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl"] Jul 10 22:23:36.590828 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:36.590801 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8d7197_6c1f_4feb_a515_fded7e132782.slice/crio-8a3b94ab6d53798d2b34fc88c80d6a82c54ed96645ad01ebaca1e0fb8b5542cf WatchSource:0}: Error finding container 8a3b94ab6d53798d2b34fc88c80d6a82c54ed96645ad01ebaca1e0fb8b5542cf: Status 404 returned error can't find the container with id 8a3b94ab6d53798d2b34fc88c80d6a82c54ed96645ad01ebaca1e0fb8b5542cf Jul 10 22:23:36.983549 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.983174 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 22:23:36.997152 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.997124 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:36.999874 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:36.999850 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 22:23:37.000075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.000058 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 22:23:37.000268 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.000251 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 22:23:37.000614 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.000589 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 22:23:37.000697 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.000653 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 22:23:37.000762 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.000716 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 22:23:37.000861 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.000843 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 22:23:37.001574 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.001198 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6piuha41ioeac\"" Jul 10 22:23:37.001574 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.001082 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 22:23:37.001574 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.001027 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 22:23:37.001574 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.001142 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 22:23:37.001832 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.001783 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 22:23:37.002329 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.002100 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r2wr9\"" Jul 10 22:23:37.002329 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.002133 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 22:23:37.004077 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.003948 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 22:23:37.004426 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.004409 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 22:23:37.009855 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.009831 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 22:23:37.048019 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.047976 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048019 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048010 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048146 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048146 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048102 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048146 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048138 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048296 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048169 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048296 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s4q9\" (UniqueName: \"kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-kube-api-access-7s4q9\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048296 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048231 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048459 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048310 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048459 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048336 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048459 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048358 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048459 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048380 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048459 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048459 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048434 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048745 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048479 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048745 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048496 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048745 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048526 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048745 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048745 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048580 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.048745 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.048600 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149379 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149294 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149379 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149344 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149591 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149392 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149591 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149591 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149457 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149591 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149485 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s4q9\" (UniqueName: \"kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-kube-api-access-7s4q9\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149591 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149528 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149591 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149759 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149599 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149759 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149759 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149654 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149759 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149759 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149759 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149732 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.149759 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149756 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.150053 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149803 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.150053 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.150053 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149868 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.150053 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.150053 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.149922 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.150610 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.150575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.152966 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.151624 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.152966 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.151700 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.152966 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.152015 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.152966 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.152656 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.153821 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.153796 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.154026 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.154003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.154332 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.154307 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.155719 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.155676 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.155885 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.155864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.156065 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.155974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.156475 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.156418 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config-out\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.157476 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.157068 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.157476 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.157089 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.157476 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.157165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.157668 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.157531 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-web-config\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.157777 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.157738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.158158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.158137 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.159263 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.159237 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s4q9\" (UniqueName: \"kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-kube-api-access-7s4q9\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.159781 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.159740 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.205707 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.205667 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" event={"ID":"1d8d7197-6c1f-4feb-a515-fded7e132782","Type":"ContainerStarted","Data":"8a3b94ab6d53798d2b34fc88c80d6a82c54ed96645ad01ebaca1e0fb8b5542cf"} Jul 10 22:23:37.207347 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.207315 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" event={"ID":"0290b83d-bf15-4cc2-aac0-7daeaaf40227","Type":"ContainerStarted","Data":"f50544b715abcadae0803208ea70f7d8f509d8cfbe2fb97ef6aad6edca282d8b"} Jul 10 22:23:37.211150 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.210967 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" event={"ID":"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff","Type":"ContainerStarted","Data":"69ea95b90f6e1c89627262c0c1d75775b698ff5b2ad6be6d4041117661e52b15"} Jul 10 22:23:37.211150 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.211052 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" event={"ID":"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff","Type":"ContainerStarted","Data":"42f173befdb5d17e09e4f0603859111243ee805653b71f6fe79b633d4454cd48"} Jul 10 22:23:37.211150 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.211101 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" event={"ID":"0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff","Type":"ContainerStarted","Data":"712a518614eecb908bad2bd3fa9f361205722c18b69e3fb44b669420582a2322"} Jul 10 22:23:37.238813 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.238764 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" podStartSLOduration=1.7421853870000001 podStartE2EDuration="5.238748141s" totalImagesPullingTime="3.496562754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:23:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:37.237560754 +0000 UTC m=+105.085842628" watchObservedRunningTime="2026-07-10 22:23:37.238748141 +0000 UTC m=+105.087030018" Jul 10 22:23:37.312931 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.312728 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:37.832103 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:37.831712 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 22:23:38.176387 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:23:38.176348 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a285cd_cc13_4b78_ae49_be2302bc1a2f.slice/crio-73f837cd471376b2fda1aeb6e35a3fd47a0477b53823413b2500457062832bc5 WatchSource:0}: Error finding container 73f837cd471376b2fda1aeb6e35a3fd47a0477b53823413b2500457062832bc5: Status 404 returned error can't find the container with id 73f837cd471376b2fda1aeb6e35a3fd47a0477b53823413b2500457062832bc5 Jul 10 22:23:38.217920 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:38.217888 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerStarted","Data":"73f837cd471376b2fda1aeb6e35a3fd47a0477b53823413b2500457062832bc5"} Jul 10 22:23:38.219871 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:38.219760 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" event={"ID":"c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec","Type":"ContainerStarted","Data":"49250f3a511ef99a4c0a6587bd4c55853861ede0ca0301d32963e4f6de9e0c4e"} Jul 10 22:23:38.220129 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:38.220105 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:38.242125 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:38.242069 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" podStartSLOduration=1.411270326 podStartE2EDuration="3.242049213s" totalImagesPullingTime="1.830778887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:38.240594121 +0000 UTC m=+106.088875997" watchObservedRunningTime="2026-07-10 22:23:38.242049213 +0000 UTC m=+106.090331092" Jul 10 22:23:38.810291 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:38.810242 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7db6cd9c68-dft8m"] Jul 10 22:23:39.224774 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:39.224750 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" event={"ID":"1d8d7197-6c1f-4feb-a515-fded7e132782","Type":"ContainerStarted","Data":"c36fc875803e52e61f2e5834fec289c9975906db29963197287018c3ec020a1a"} Jul 10 22:23:39.225021 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:39.224963 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" Jul 10 22:23:39.226833 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:39.226811 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" event={"ID":"0290b83d-bf15-4cc2-aac0-7daeaaf40227","Type":"ContainerStarted","Data":"65ee757ad3b63f550f1c5c325f2dee71126154bbee2f5c10362004a64359cba8"} Jul 10 22:23:39.230793 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:39.230769 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" Jul 10 22:23:39.239638 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:39.239600 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-vn2nl" podStartSLOduration=2.189899287 podStartE2EDuration="4.239587219s" totalImagesPullingTime="2.049687932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:39.238383676 +0000 UTC m=+107.086665553" watchObservedRunningTime="2026-07-10 22:23:39.239587219 +0000 UTC m=+107.087869093" Jul 10 22:23:40.232737 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:40.232699 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" event={"ID":"0290b83d-bf15-4cc2-aac0-7daeaaf40227","Type":"ContainerStarted","Data":"bba26be1c9a55b0d0a0752c188eb811112432b8a74faa3bf55e215d9ec6333d7"} Jul 10 22:23:40.233179 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:40.232762 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" event={"ID":"0290b83d-bf15-4cc2-aac0-7daeaaf40227","Type":"ContainerStarted","Data":"e6ea4cc09bbc4219d2c5e78b684da503979cb095888f53d807382406d185e933"} Jul 10 22:23:40.234146 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:40.234121 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerID="575e84f55968dc08b87ba91069ccaa05545da1e3f441cfbdd9d6809c66fe0c10" exitCode=0 Jul 10 22:23:40.234286 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:40.234220 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerDied","Data":"575e84f55968dc08b87ba91069ccaa05545da1e3f441cfbdd9d6809c66fe0c10"} Jul 10 22:23:40.253807 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:40.253759 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-f44675f6-c7lbg" podStartSLOduration=2.566881649 podStartE2EDuration="5.253743645s" totalImagesPullingTime="2.686861996s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:23:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:40.253469301 +0000 UTC m=+108.101751177" watchObservedRunningTime="2026-07-10 22:23:40.253743645 +0000 UTC m=+108.102025520" Jul 10 22:23:44.233683 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:44.233650 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86b8dff754-cxzkd" Jul 10 22:23:44.257052 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:44.257019 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerStarted","Data":"0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f"} Jul 10 22:23:44.257202 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:44.257097 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerStarted","Data":"055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2"} Jul 10 22:23:44.257202 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:44.257149 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerStarted","Data":"b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7"} Jul 10 22:23:44.257202 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:44.257188 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerStarted","Data":"fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f"} Jul 10 22:23:44.257314 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:44.257225 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerStarted","Data":"0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0"} Jul 10 22:23:44.257314 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:44.257260 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerStarted","Data":"8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f"} Jul 10 22:23:44.285730 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:44.285680 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.984191554 podStartE2EDuration="8.285663314s" totalImagesPullingTime="4.284046298s" totalInitContainerRuntime="17.425462ms" isStatefulPod=false podCreationTimestamp="2026-07-10 22:23:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:23:44.28442705 +0000 UTC m=+112.132708948" watchObservedRunningTime="2026-07-10 22:23:44.285663314 +0000 UTC m=+112.133945188" Jul 10 22:23:46.213752 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:46.213722 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8s7sv" Jul 10 22:23:47.313804 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:47.313772 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:23:48.816755 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:48.816730 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:23:55.420418 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:55.420385 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:23:55.420963 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:23:55.420460 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:24:02.572625 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:02.572589 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:24:02.574830 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:02.574812 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/81ca9b9c-e95f-4ac1-9e97-96169222847b-metrics-certs\") pod \"network-metrics-daemon-wbxsz\" (UID: \"81ca9b9c-e95f-4ac1-9e97-96169222847b\") " pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:24:02.631947 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:02.631917 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qbckj\"" Jul 10 22:24:02.640605 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:02.640580 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wbxsz" Jul 10 22:24:02.773998 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:02.773970 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wbxsz"] Jul 10 22:24:02.776484 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:24:02.776458 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ca9b9c_e95f_4ac1_9e97_96169222847b.slice/crio-73300075a45816198fc4f8e60ca330f7974548f6bcd94621990bdcca13d546dc WatchSource:0}: Error finding container 73300075a45816198fc4f8e60ca330f7974548f6bcd94621990bdcca13d546dc: Status 404 returned error can't find the container with id 73300075a45816198fc4f8e60ca330f7974548f6bcd94621990bdcca13d546dc Jul 10 22:24:03.331383 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:03.331346 2559 generic.go:356] "Generic (PLEG): container finished" podID="2be24f5f-4532-4c5f-bebf-5c5b76a699cd" containerID="c8bf53378d26205cd446798ad67bfeb571d21b887f75bc2f06030037d13d7355" exitCode=0 Jul 10 22:24:03.331607 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:03.331462 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" event={"ID":"2be24f5f-4532-4c5f-bebf-5c5b76a699cd","Type":"ContainerDied","Data":"c8bf53378d26205cd446798ad67bfeb571d21b887f75bc2f06030037d13d7355"} Jul 10 22:24:03.331914 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:03.331877 2559 scope.go:117] "RemoveContainer" containerID="c8bf53378d26205cd446798ad67bfeb571d21b887f75bc2f06030037d13d7355" Jul 10 22:24:03.333047 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:03.333019 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbxsz" event={"ID":"81ca9b9c-e95f-4ac1-9e97-96169222847b","Type":"ContainerStarted","Data":"73300075a45816198fc4f8e60ca330f7974548f6bcd94621990bdcca13d546dc"} Jul 10 22:24:03.834009 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:03.833970 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" podUID="a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" containerName="registry" containerID="cri-o://5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d" gracePeriod=30 Jul 10 22:24:04.086577 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.086554 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:24:04.183976 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.183947 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4t2t\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-kube-api-access-p4t2t\") pod \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " Jul 10 22:24:04.184158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.183986 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-image-registry-private-configuration\") pod \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " Jul 10 22:24:04.184158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.184011 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-installation-pull-secrets\") pod \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " Jul 10 22:24:04.184158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.184049 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-bound-sa-token\") pod \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " Jul 10 22:24:04.184158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.184068 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-trusted-ca\") pod \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " Jul 10 22:24:04.184158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.184085 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-certificates\") pod \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " Jul 10 22:24:04.184158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.184112 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-ca-trust-extracted\") pod \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " Jul 10 22:24:04.184158 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.184155 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") pod \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\" (UID: \"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4\") " Jul 10 22:24:04.185059 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.184829 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:04.185059 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.184695 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:04.186676 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.186643 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-kube-api-access-p4t2t" (OuterVolumeSpecName: "kube-api-access-p4t2t") pod "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4"). InnerVolumeSpecName "kube-api-access-p4t2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:24:04.186817 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.186795 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:04.186885 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.186835 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:24:04.186885 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.186847 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:04.186885 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.186846 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:24:04.195809 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.195778 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" (UID: "a41d9dd7-0403-43e9-9ea9-11e18f8d32a4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:24:04.284857 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.284775 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-bound-sa-token\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:04.284857 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.284800 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-trusted-ca\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:04.284857 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.284811 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-certificates\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:04.284857 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.284823 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-ca-trust-extracted\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:04.284857 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.284834 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-registry-tls\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:04.284857 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.284846 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4t2t\" (UniqueName: \"kubernetes.io/projected/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-kube-api-access-p4t2t\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:04.284857 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.284860 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-image-registry-private-configuration\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:04.285194 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.284873 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4-installation-pull-secrets\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:04.338943 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.338908 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbxsz" event={"ID":"81ca9b9c-e95f-4ac1-9e97-96169222847b","Type":"ContainerStarted","Data":"2400e599afd5d3efc657f1f77d2306d27cb77edb91f4adaa85ed1785148c098b"} Jul 10 22:24:04.339100 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.338963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wbxsz" event={"ID":"81ca9b9c-e95f-4ac1-9e97-96169222847b","Type":"ContainerStarted","Data":"f8c6ef0c8a97faa69b80f72f4936119c5be3a3272f22d80511a691c91b60fd52"} Jul 10 22:24:04.340120 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.340095 2559 generic.go:356] "Generic (PLEG): container finished" podID="a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" containerID="5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d" exitCode=0 Jul 10 22:24:04.340242 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.340133 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" Jul 10 22:24:04.340242 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.340171 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" event={"ID":"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4","Type":"ContainerDied","Data":"5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d"} Jul 10 22:24:04.340242 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.340212 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db6cd9c68-dft8m" event={"ID":"a41d9dd7-0403-43e9-9ea9-11e18f8d32a4","Type":"ContainerDied","Data":"604f13d759da6c1dd32238cbb3e5060c9de5ccaf8a79aef9f7446a3e62ed9f47"} Jul 10 22:24:04.340242 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.340231 2559 scope.go:117] "RemoveContainer" containerID="5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d" Jul 10 22:24:04.342010 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.341991 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-mtt7p" event={"ID":"2be24f5f-4532-4c5f-bebf-5c5b76a699cd","Type":"ContainerStarted","Data":"f809cbd94081e75e4ecfce6e0a91ae3d93a943c9aff02f0957bcb081bfa10bdd"} Jul 10 22:24:04.349371 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.349356 2559 scope.go:117] "RemoveContainer" containerID="5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d" Jul 10 22:24:04.349629 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:24:04.349614 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d\": container with ID starting with 5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d not found: ID does not exist" containerID="5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d" Jul 10 22:24:04.349684 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.349636 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d"} err="failed to get container status \"5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d\": rpc error: code = NotFound desc = could not find container \"5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d\": container with ID starting with 5f6ec170066343b73085a42405c3fa83693391b655e52a6b92626a9e0dfa876d not found: ID does not exist" Jul 10 22:24:04.357132 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.357088 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wbxsz" podStartSLOduration=130.440980954 podStartE2EDuration="2m11.35707346s" totalImagesPullingTime="916.092506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:21:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:24:04.35614407 +0000 UTC m=+132.204425947" watchObservedRunningTime="2026-07-10 22:24:04.35707346 +0000 UTC m=+132.205355335" Jul 10 22:24:04.389829 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.389805 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7db6cd9c68-dft8m"] Jul 10 22:24:04.393582 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.393559 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7db6cd9c68-dft8m"] Jul 10 22:24:04.725288 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:04.725263 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" path="/var/lib/kubelet/pods/a41d9dd7-0403-43e9-9ea9-11e18f8d32a4/volumes" Jul 10 22:24:15.425650 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:15.425622 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:24:15.429296 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:15.429277 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6d4999c9b9-mb5wx" Jul 10 22:24:19.406076 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:19.406039 2559 generic.go:356] "Generic (PLEG): container finished" podID="10b1ab46-05e5-413a-9c4e-5abde8f71eed" containerID="b9237d1cdbd1c5870ded7d4fac1bb4defb65afbe8f68255d704718887c262eac" exitCode=0 Jul 10 22:24:19.406578 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:19.406098 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" event={"ID":"10b1ab46-05e5-413a-9c4e-5abde8f71eed","Type":"ContainerDied","Data":"b9237d1cdbd1c5870ded7d4fac1bb4defb65afbe8f68255d704718887c262eac"} Jul 10 22:24:19.406578 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:19.406544 2559 scope.go:117] "RemoveContainer" containerID="b9237d1cdbd1c5870ded7d4fac1bb4defb65afbe8f68255d704718887c262eac" Jul 10 22:24:20.412749 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:20.412707 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-g9djw" event={"ID":"10b1ab46-05e5-413a-9c4e-5abde8f71eed","Type":"ContainerStarted","Data":"2d0b05490292f710794b2e8a64e1b49b4e5ff5bc2513fae7cde96011fdad54a1"} Jul 10 22:24:23.426537 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:23.426506 2559 generic.go:356] "Generic (PLEG): container finished" podID="64cfd78b-f6a5-4da0-ab73-dd6928c55dae" containerID="8e24d91f59755d0cb41820ce6bd08a258328c86d9b703b8ee8f6e2419f15e65c" exitCode=0 Jul 10 22:24:23.426890 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:23.426550 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" event={"ID":"64cfd78b-f6a5-4da0-ab73-dd6928c55dae","Type":"ContainerDied","Data":"8e24d91f59755d0cb41820ce6bd08a258328c86d9b703b8ee8f6e2419f15e65c"} Jul 10 22:24:23.426890 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:23.426855 2559 scope.go:117] "RemoveContainer" containerID="8e24d91f59755d0cb41820ce6bd08a258328c86d9b703b8ee8f6e2419f15e65c" Jul 10 22:24:24.431297 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:24.431266 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s7t6h_8ad3159f-50bb-4fc7-af39-e85508a1575d/serve-healthcheck-canary/0.log" Jul 10 22:24:24.432887 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:24.432861 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-k8jhp" event={"ID":"64cfd78b-f6a5-4da0-ab73-dd6928c55dae","Type":"ContainerStarted","Data":"3c45da6be223911ba9f1324739d3ee3b92b078de451c3ec0e1008a6ae8407714"} Jul 10 22:24:37.313472 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:37.313416 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:37.341384 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:37.341352 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:37.504245 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:37.504218 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:55.332391 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.332360 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 22:24:55.332828 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.332645 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="prometheus" containerID="cri-o://8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f" gracePeriod=600 Jul 10 22:24:55.332828 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.332675 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy" containerID="cri-o://055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2" gracePeriod=600 Jul 10 22:24:55.332828 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.332705 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="thanos-sidecar" containerID="cri-o://fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f" gracePeriod=600 Jul 10 22:24:55.332828 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.332739 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="config-reloader" containerID="cri-o://0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0" gracePeriod=600 Jul 10 22:24:55.332828 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.332707 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f" gracePeriod=600 Jul 10 22:24:55.332828 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.332705 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy-web" containerID="cri-o://b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7" gracePeriod=600 Jul 10 22:24:55.576153 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.576122 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerID="0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f" exitCode=0 Jul 10 22:24:55.576153 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.576149 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerID="055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2" exitCode=0 Jul 10 22:24:55.576153 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.576156 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerID="fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f" exitCode=0 Jul 10 22:24:55.576153 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.576161 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerID="0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0" exitCode=0 Jul 10 22:24:55.576387 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.576167 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerID="8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f" exitCode=0 Jul 10 22:24:55.576387 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.576193 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerDied","Data":"0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f"} Jul 10 22:24:55.576387 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.576281 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerDied","Data":"055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2"} Jul 10 22:24:55.576387 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.576309 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerDied","Data":"fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f"} Jul 10 22:24:55.576387 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.576340 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerDied","Data":"0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0"} Jul 10 22:24:55.576387 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:55.576363 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerDied","Data":"8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f"} Jul 10 22:24:56.577304 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.577282 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:56.583256 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.583219 2559 generic.go:356] "Generic (PLEG): container finished" podID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerID="b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7" exitCode=0 Jul 10 22:24:56.583361 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.583299 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:56.583404 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.583296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerDied","Data":"b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7"} Jul 10 22:24:56.583434 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.583424 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b4a285cd-cc13-4b78-ae49-be2302bc1a2f","Type":"ContainerDied","Data":"73f837cd471376b2fda1aeb6e35a3fd47a0477b53823413b2500457062832bc5"} Jul 10 22:24:56.583489 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.583463 2559 scope.go:117] "RemoveContainer" containerID="0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f" Jul 10 22:24:56.590652 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.590634 2559 scope.go:117] "RemoveContainer" containerID="055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2" Jul 10 22:24:56.598652 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.598623 2559 scope.go:117] "RemoveContainer" containerID="b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7" Jul 10 22:24:56.606585 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.606560 2559 scope.go:117] "RemoveContainer" containerID="fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f" Jul 10 22:24:56.615942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.615921 2559 scope.go:117] "RemoveContainer" containerID="0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0" Jul 10 22:24:56.624268 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.624251 2559 scope.go:117] "RemoveContainer" containerID="8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f" Jul 10 22:24:56.630476 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630455 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7s4q9\" (UniqueName: \"kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-kube-api-access-7s4q9\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.630584 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630499 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.630584 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630536 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-grpc-tls\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.630753 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630625 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-0\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.630753 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630660 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-trusted-ca-bundle\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.630753 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630710 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-kube-rbac-proxy\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.630753 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630748 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-1\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630775 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config-out\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630806 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-thanos-prometheus-http-client-file\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630850 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630877 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630910 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-tls-assets\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630950 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-db\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630974 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-tls\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.630998 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-serving-certs-ca-bundle\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.631055 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-kubelet-serving-ca-bundle\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631546 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.631089 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-web-config\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631546 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.631118 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-metrics-client-ca\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631546 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.631129 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:56.631546 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.631157 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-metrics-client-certs\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631546 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.631193 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-2\") pod \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\" (UID: \"b4a285cd-cc13-4b78-ae49-be2302bc1a2f\") " Jul 10 22:24:56.631546 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.631364 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:56.631811 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.631552 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.631811 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.631662 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.631811 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.631715 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:56.632246 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.632020 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:56.632574 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.632360 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:56.632574 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.632494 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:56.634692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.634662 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:24:56.634892 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.634861 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 22:24:56.635179 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.635144 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config-out" (OuterVolumeSpecName: "config-out") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:24:56.635179 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.635162 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:56.635858 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.635832 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:56.635952 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.635878 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:56.635952 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.635919 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config" (OuterVolumeSpecName: "config") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:56.636134 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.636064 2559 scope.go:117] "RemoveContainer" containerID="575e84f55968dc08b87ba91069ccaa05545da1e3f441cfbdd9d6809c66fe0c10" Jul 10 22:24:56.636192 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.636151 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:56.636537 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.636495 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-kube-api-access-7s4q9" (OuterVolumeSpecName: "kube-api-access-7s4q9") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "kube-api-access-7s4q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:24:56.636760 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.636727 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:56.636851 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.636756 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:56.637631 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.637607 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:24:56.638022 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.637989 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:56.649301 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.649280 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-web-config" (OuterVolumeSpecName: "web-config") pod "b4a285cd-cc13-4b78-ae49-be2302bc1a2f" (UID: "b4a285cd-cc13-4b78-ae49-be2302bc1a2f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:24:56.654194 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.654171 2559 scope.go:117] "RemoveContainer" containerID="0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f" Jul 10 22:24:56.654498 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:24:56.654475 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f\": container with ID starting with 0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f not found: ID does not exist" containerID="0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f" Jul 10 22:24:56.654579 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.654505 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f"} err="failed to get container status \"0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f\": rpc error: code = NotFound desc = could not find container \"0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f\": container with ID starting with 0580b1e08c0be17d9ffcfd62960e1a85b7b046f827f6067e7fff52ee438cc88f not found: ID does not exist" Jul 10 22:24:56.654579 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.654523 2559 scope.go:117] "RemoveContainer" containerID="055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2" Jul 10 22:24:56.654801 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:24:56.654779 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2\": container with ID starting with 055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2 not found: ID does not exist" containerID="055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2" Jul 10 22:24:56.654847 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.654809 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2"} err="failed to get container status \"055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2\": rpc error: code = NotFound desc = could not find container \"055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2\": container with ID starting with 055f044ce38978e2c49b6813920cf6cdc0acd3cd3a4ce04b1a6620d01953def2 not found: ID does not exist" Jul 10 22:24:56.654847 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.654826 2559 scope.go:117] "RemoveContainer" containerID="b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7" Jul 10 22:24:56.655053 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:24:56.655035 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7\": container with ID starting with b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7 not found: ID does not exist" containerID="b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7" Jul 10 22:24:56.655095 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.655059 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7"} err="failed to get container status \"b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7\": rpc error: code = NotFound desc = could not find container \"b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7\": container with ID starting with b5d92aba2b2ec7f9cfe55cca23ebd02385d7d1485609b2a07962a530f01e4fb7 not found: ID does not exist" Jul 10 22:24:56.655095 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.655074 2559 scope.go:117] "RemoveContainer" containerID="fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f" Jul 10 22:24:56.655310 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:24:56.655292 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f\": container with ID starting with fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f not found: ID does not exist" containerID="fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f" Jul 10 22:24:56.655373 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.655317 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f"} err="failed to get container status \"fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f\": rpc error: code = NotFound desc = could not find container \"fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f\": container with ID starting with fe11a8b7a54b401e1f5b7ef94071cafe5836cab62d37787df31019484409aa4f not found: ID does not exist" Jul 10 22:24:56.655373 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.655337 2559 scope.go:117] "RemoveContainer" containerID="0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0" Jul 10 22:24:56.655621 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:24:56.655599 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0\": container with ID starting with 0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0 not found: ID does not exist" containerID="0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0" Jul 10 22:24:56.655674 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.655627 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0"} err="failed to get container status \"0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0\": rpc error: code = NotFound desc = could not find container \"0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0\": container with ID starting with 0bd06667c6f30dc372af53ec7e4c37281e9fc229b4c0506572cf68f2ee136eb0 not found: ID does not exist" Jul 10 22:24:56.655674 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.655647 2559 scope.go:117] "RemoveContainer" containerID="8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f" Jul 10 22:24:56.655890 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:24:56.655872 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f\": container with ID starting with 8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f not found: ID does not exist" containerID="8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f" Jul 10 22:24:56.655931 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.655895 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f"} err="failed to get container status \"8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f\": rpc error: code = NotFound desc = could not find container \"8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f\": container with ID starting with 8bd88436834baba8eeca47ad65cc1aef114db766d0c51aa7f9ad5d6e313b454f not found: ID does not exist" Jul 10 22:24:56.655931 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.655913 2559 scope.go:117] "RemoveContainer" containerID="575e84f55968dc08b87ba91069ccaa05545da1e3f441cfbdd9d6809c66fe0c10" Jul 10 22:24:56.656171 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:24:56.656154 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"575e84f55968dc08b87ba91069ccaa05545da1e3f441cfbdd9d6809c66fe0c10\": container with ID starting with 575e84f55968dc08b87ba91069ccaa05545da1e3f441cfbdd9d6809c66fe0c10 not found: ID does not exist" containerID="575e84f55968dc08b87ba91069ccaa05545da1e3f441cfbdd9d6809c66fe0c10" Jul 10 22:24:56.656216 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.656175 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"575e84f55968dc08b87ba91069ccaa05545da1e3f441cfbdd9d6809c66fe0c10"} err="failed to get container status \"575e84f55968dc08b87ba91069ccaa05545da1e3f441cfbdd9d6809c66fe0c10\": rpc error: code = NotFound desc = could not find container \"575e84f55968dc08b87ba91069ccaa05545da1e3f441cfbdd9d6809c66fe0c10\": container with ID starting with 575e84f55968dc08b87ba91069ccaa05545da1e3f441cfbdd9d6809c66fe0c10 not found: ID does not exist" Jul 10 22:24:56.732924 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.732899 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.732924 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.732924 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s4q9\" (UniqueName: \"kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-kube-api-access-7s4q9\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.732939 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.732953 2559 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-grpc-tls\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.732966 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.732979 2559 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-kube-rbac-proxy\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.732991 2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config-out\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.733004 2559 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.733016 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.733029 2559 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-config\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.733043 2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-tls-assets\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.733055 2559 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-prometheus-k8s-db\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.733068 2559 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.733082 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733107 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.733097 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733516 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.733112 2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-web-config\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733516 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.733126 2559 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-configmap-metrics-client-ca\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.733516 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.733143 2559 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b4a285cd-cc13-4b78-ae49-be2302bc1a2f-secret-metrics-client-certs\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:24:56.902428 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.902390 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 22:24:56.905688 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.905620 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 22:24:56.933372 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933343 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 22:24:56.933657 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933643 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="prometheus" Jul 10 22:24:56.933702 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933659 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="prometheus" Jul 10 22:24:56.933702 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933669 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" containerName="registry" Jul 10 22:24:56.933702 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933674 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" containerName="registry" Jul 10 22:24:56.933702 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933681 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="init-config-reloader" Jul 10 22:24:56.933702 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933687 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="init-config-reloader" Jul 10 22:24:56.933702 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933692 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="thanos-sidecar" Jul 10 22:24:56.933702 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933697 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="thanos-sidecar" Jul 10 22:24:56.933702 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933703 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy-thanos" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933708 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy-thanos" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933717 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933723 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933729 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="config-reloader" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933735 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="config-reloader" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933741 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy-web" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933745 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy-web" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933803 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="config-reloader" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933813 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933821 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a41d9dd7-0403-43e9-9ea9-11e18f8d32a4" containerName="registry" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933826 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="thanos-sidecar" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933832 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="prometheus" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933837 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy-web" Jul 10 22:24:56.933942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.933841 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" containerName="kube-rbac-proxy-thanos" Jul 10 22:24:56.938918 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.938901 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:56.941155 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.941135 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 10 22:24:56.941254 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.941235 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 10 22:24:56.941378 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.941363 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 10 22:24:56.941487 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.941472 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 10 22:24:56.941560 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.941543 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 10 22:24:56.941869 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.941850 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-r2wr9\"" Jul 10 22:24:56.941984 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.941969 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 10 22:24:56.941984 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.941980 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 10 22:24:56.942349 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.942322 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6piuha41ioeac\"" Jul 10 22:24:56.942412 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.942348 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 10 22:24:56.942412 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.942360 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 10 22:24:56.942412 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.942372 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 10 22:24:56.942412 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.942336 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 10 22:24:56.942710 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.942696 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 10 22:24:56.945980 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.945960 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 10 22:24:56.948665 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.948646 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 10 22:24:56.954923 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:56.954902 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 22:24:57.036005 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.035974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036005 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036007 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-web-config\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036191 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036026 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036191 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036046 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036191 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036066 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036191 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036084 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dac91be4-3c64-4c2b-9617-aec2bfd5c512-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036191 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036191 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mhf5\" (UniqueName: \"kubernetes.io/projected/dac91be4-3c64-4c2b-9617-aec2bfd5c512-kube-api-access-8mhf5\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036521 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036219 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036521 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036265 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036521 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036297 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036521 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036323 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036521 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036361 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-config\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036521 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036521 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036459 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036521 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036800 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036537 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036800 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036562 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036800 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036587 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.036800 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.036642 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dac91be4-3c64-4c2b-9617-aec2bfd5c512-config-out\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.137936 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.137909 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.137936 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.137939 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-config\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138153 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.137962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138153 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.137983 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138153 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138024 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138153 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138153 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138096 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138153 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138120 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138153 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138147 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dac91be4-3c64-4c2b-9617-aec2bfd5c512-config-out\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138172 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138203 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-web-config\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138232 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138257 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dac91be4-3c64-4c2b-9617-aec2bfd5c512-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mhf5\" (UniqueName: \"kubernetes.io/projected/dac91be4-3c64-4c2b-9617-aec2bfd5c512-kube-api-access-8mhf5\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138410 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138470 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.138515 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.138504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.139926 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.139208 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.139926 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.139464 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.139926 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.139588 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.139926 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.139591 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.140254 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.140229 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.140770 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.140737 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.141000 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.140979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.141064 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.140982 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-config\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.141129 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.141105 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.141552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.141530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.141987 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.141962 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/dac91be4-3c64-4c2b-9617-aec2bfd5c512-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.142387 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.142360 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.142510 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.142411 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.142578 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.142532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.143345 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.143317 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/dac91be4-3c64-4c2b-9617-aec2bfd5c512-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.143475 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.143460 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.143540 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.143519 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.143711 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.143697 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/dac91be4-3c64-4c2b-9617-aec2bfd5c512-config-out\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.144297 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.144282 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/dac91be4-3c64-4c2b-9617-aec2bfd5c512-web-config\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.147546 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.147525 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mhf5\" (UniqueName: \"kubernetes.io/projected/dac91be4-3c64-4c2b-9617-aec2bfd5c512-kube-api-access-8mhf5\") pod \"prometheus-k8s-0\" (UID: \"dac91be4-3c64-4c2b-9617-aec2bfd5c512\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.249151 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.249091 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:24:57.386712 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.386691 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 10 22:24:57.388873 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:24:57.388845 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac91be4_3c64_4c2b_9617_aec2bfd5c512.slice/crio-122628a003118c901e46f82469074d3426daf3160a77f1b585c1d465b680a72f WatchSource:0}: Error finding container 122628a003118c901e46f82469074d3426daf3160a77f1b585c1d465b680a72f: Status 404 returned error can't find the container with id 122628a003118c901e46f82469074d3426daf3160a77f1b585c1d465b680a72f Jul 10 22:24:57.589875 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.589799 2559 generic.go:356] "Generic (PLEG): container finished" podID="dac91be4-3c64-4c2b-9617-aec2bfd5c512" containerID="d9b4e427f792e5eb37849f0109b1b60fe6d377312d3c4a81db200ca7ef2cc330" exitCode=0 Jul 10 22:24:57.589875 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.589839 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dac91be4-3c64-4c2b-9617-aec2bfd5c512","Type":"ContainerDied","Data":"d9b4e427f792e5eb37849f0109b1b60fe6d377312d3c4a81db200ca7ef2cc330"} Jul 10 22:24:57.590343 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:57.589887 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dac91be4-3c64-4c2b-9617-aec2bfd5c512","Type":"ContainerStarted","Data":"122628a003118c901e46f82469074d3426daf3160a77f1b585c1d465b680a72f"} Jul 10 22:24:58.597459 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:58.597406 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dac91be4-3c64-4c2b-9617-aec2bfd5c512","Type":"ContainerStarted","Data":"38adabd1e4eb035f8cf6bc865f9d76bbf4c6421d737f2bec7f5b86e6231aff34"} Jul 10 22:24:58.597921 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:58.597546 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dac91be4-3c64-4c2b-9617-aec2bfd5c512","Type":"ContainerStarted","Data":"afcbb6597825ce26a4929dfb448730ec765bd9c374a4b0fa6dc6f1f207b4ec06"} Jul 10 22:24:58.597921 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:58.597598 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dac91be4-3c64-4c2b-9617-aec2bfd5c512","Type":"ContainerStarted","Data":"dd6ae7f21b0ab9add578fc9d3f132ab4f063a5e806319678508827a1fa131214"} Jul 10 22:24:58.597921 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:58.597640 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dac91be4-3c64-4c2b-9617-aec2bfd5c512","Type":"ContainerStarted","Data":"da45215bc744928f54488c3e14e0e42c8ef7b0b0d9b58f7a7bfd0baf2f24f69c"} Jul 10 22:24:58.597921 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:58.597679 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dac91be4-3c64-4c2b-9617-aec2bfd5c512","Type":"ContainerStarted","Data":"92d7314264d0cda8805ad78e33df38446d7d4a99fd7bd10796c987d85a0331ac"} Jul 10 22:24:58.597921 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:58.597718 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"dac91be4-3c64-4c2b-9617-aec2bfd5c512","Type":"ContainerStarted","Data":"a84ffa8f00a0e14e194770d43dbb40123d62528a2e932a6b31b77b2274348c6c"} Jul 10 22:24:58.627219 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:58.627172 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.608324776 podStartE2EDuration="2.627155879s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.831103ms" isStatefulPod=false podCreationTimestamp="2026-07-10 22:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:24:58.625955209 +0000 UTC m=+186.474237084" watchObservedRunningTime="2026-07-10 22:24:58.627155879 +0000 UTC m=+186.475437756" Jul 10 22:24:58.725061 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:24:58.725028 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a285cd-cc13-4b78-ae49-be2302bc1a2f" path="/var/lib/kubelet/pods/b4a285cd-cc13-4b78-ae49-be2302bc1a2f/volumes" Jul 10 22:25:02.249552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:25:02.249516 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:25:57.249250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:25:57.249214 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:25:57.264539 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:25:57.264508 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:25:57.856544 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:25:57.856513 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 10 22:26:34.792721 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.792693 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x"] Jul 10 22:26:34.796130 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.796109 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:34.798811 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.798789 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 22:26:34.799133 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.799117 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 22:26:34.799245 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.799225 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hzgd7\"" Jul 10 22:26:34.799453 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.799427 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 22:26:34.799612 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.799596 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 22:26:34.808677 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.808654 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x"] Jul 10 22:26:34.946127 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.946066 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8t8f\" (UniqueName: \"kubernetes.io/projected/27f48dab-3d0c-4ea9-a416-bcc8c659a938-kube-api-access-t8t8f\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:34.946127 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.946134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f48dab-3d0c-4ea9-a416-bcc8c659a938-tmp\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:34.946388 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.946203 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27f48dab-3d0c-4ea9-a416-bcc8c659a938-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:34.946388 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.946271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f48dab-3d0c-4ea9-a416-bcc8c659a938-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:34.946388 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:34.946319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27f48dab-3d0c-4ea9-a416-bcc8c659a938-manifests\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.047763 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.047673 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27f48dab-3d0c-4ea9-a416-bcc8c659a938-manifests\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.047763 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.047737 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8t8f\" (UniqueName: \"kubernetes.io/projected/27f48dab-3d0c-4ea9-a416-bcc8c659a938-kube-api-access-t8t8f\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.047942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.047776 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f48dab-3d0c-4ea9-a416-bcc8c659a938-tmp\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.047942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.047799 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27f48dab-3d0c-4ea9-a416-bcc8c659a938-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.048005 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.047946 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f48dab-3d0c-4ea9-a416-bcc8c659a938-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.048106 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.048090 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/27f48dab-3d0c-4ea9-a416-bcc8c659a938-manifests\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.050192 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.050161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f48dab-3d0c-4ea9-a416-bcc8c659a938-tmp\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.050367 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.050345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/27f48dab-3d0c-4ea9-a416-bcc8c659a938-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.050428 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.050345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/27f48dab-3d0c-4ea9-a416-bcc8c659a938-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.059133 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.059108 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8t8f\" (UniqueName: \"kubernetes.io/projected/27f48dab-3d0c-4ea9-a416-bcc8c659a938-kube-api-access-t8t8f\") pod \"opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x\" (UID: \"27f48dab-3d0c-4ea9-a416-bcc8c659a938\") " pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.107017 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.106989 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:35.241504 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.241476 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x"] Jul 10 22:26:35.244782 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:26:35.244759 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f48dab_3d0c_4ea9_a416_bcc8c659a938.slice/crio-1b33f38bb477164ee7d2697c9eb340b873fa3efeaf5c6f455918d241a5de9402 WatchSource:0}: Error finding container 1b33f38bb477164ee7d2697c9eb340b873fa3efeaf5c6f455918d241a5de9402: Status 404 returned error can't find the container with id 1b33f38bb477164ee7d2697c9eb340b873fa3efeaf5c6f455918d241a5de9402 Jul 10 22:26:35.994325 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:35.994287 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" event={"ID":"27f48dab-3d0c-4ea9-a416-bcc8c659a938","Type":"ContainerStarted","Data":"1b33f38bb477164ee7d2697c9eb340b873fa3efeaf5c6f455918d241a5de9402"} Jul 10 22:26:39.009916 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:39.009877 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" event={"ID":"27f48dab-3d0c-4ea9-a416-bcc8c659a938","Type":"ContainerStarted","Data":"dab4c7d7989e6c4176ad1c681436779fb4911b555b88066e91f3fb8bb2c6243d"} Jul 10 22:26:40.015638 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:40.015601 2559 generic.go:356] "Generic (PLEG): container finished" podID="27f48dab-3d0c-4ea9-a416-bcc8c659a938" containerID="dab4c7d7989e6c4176ad1c681436779fb4911b555b88066e91f3fb8bb2c6243d" exitCode=0 Jul 10 22:26:40.015998 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:40.015650 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" event={"ID":"27f48dab-3d0c-4ea9-a416-bcc8c659a938","Type":"ContainerDied","Data":"dab4c7d7989e6c4176ad1c681436779fb4911b555b88066e91f3fb8bb2c6243d"} Jul 10 22:26:41.021498 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:41.021460 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" event={"ID":"27f48dab-3d0c-4ea9-a416-bcc8c659a938","Type":"ContainerStarted","Data":"75aad6d6b9c000006841ffc10675ead2b900d2822e7531c4c9f7ad13d7837872"} Jul 10 22:26:41.021881 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:41.021560 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:41.041750 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:41.041700 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" podStartSLOduration=2.762971634 podStartE2EDuration="7.041681517s" totalImagesPullingTime="3.242514726s" totalInitContainerRuntime="1.036195157s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:26:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:26:41.040049627 +0000 UTC m=+288.888331503" watchObservedRunningTime="2026-07-10 22:26:41.041681517 +0000 UTC m=+288.889963397" Jul 10 22:26:52.028052 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:52.028025 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x" Jul 10 22:26:52.589068 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:52.589039 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:26:52.597131 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:52.597105 2559 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 22:26:52.597269 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:52.597173 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:26:54.236166 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.236132 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf"] Jul 10 22:26:54.238516 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.238498 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.245667 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.245647 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 10 22:26:54.246665 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.246644 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 10 22:26:54.246760 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.246688 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 22:26:54.246817 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.246805 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 10 22:26:54.246907 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.246891 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 22:26:54.246960 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.246910 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zjk5j\"" Jul 10 22:26:54.253360 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.253337 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf"] Jul 10 22:26:54.303203 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.303170 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-metrics-cert\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.303342 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.303232 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-manager-config\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.303342 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.303268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wk7w\" (UniqueName: \"kubernetes.io/projected/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-kube-api-access-8wk7w\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.303342 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.303298 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-cert\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.346584 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.346560 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp"] Jul 10 22:26:54.349115 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.349100 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" Jul 10 22:26:54.357383 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.357360 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 22:26:54.359058 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.359038 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 22:26:54.361929 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.361914 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 22:26:54.363301 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.363278 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 22:26:54.363510 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.363496 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-76cwd\"" Jul 10 22:26:54.367982 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.367961 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp"] Jul 10 22:26:54.403629 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.403604 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b3e68d-303d-4a4e-81e4-c62ed13a66bf-tmp\") pod \"kube-auth-proxy-fc969b7d8-zrfbp\" (UID: \"82b3e68d-303d-4a4e-81e4-c62ed13a66bf\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" Jul 10 22:26:54.403722 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.403673 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-cert\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.403722 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.403707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-metrics-cert\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.403794 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.403740 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg5qp\" (UniqueName: \"kubernetes.io/projected/82b3e68d-303d-4a4e-81e4-c62ed13a66bf-kube-api-access-kg5qp\") pod \"kube-auth-proxy-fc969b7d8-zrfbp\" (UID: \"82b3e68d-303d-4a4e-81e4-c62ed13a66bf\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" Jul 10 22:26:54.403843 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.403792 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-manager-config\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.403843 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.403819 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b3e68d-303d-4a4e-81e4-c62ed13a66bf-tls-certs\") pod \"kube-auth-proxy-fc969b7d8-zrfbp\" (UID: \"82b3e68d-303d-4a4e-81e4-c62ed13a66bf\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" Jul 10 22:26:54.403937 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.403917 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wk7w\" (UniqueName: \"kubernetes.io/projected/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-kube-api-access-8wk7w\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.404391 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.404372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-manager-config\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.406049 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.406023 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-cert\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.406210 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.406190 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-metrics-cert\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.432421 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.432392 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wk7w\" (UniqueName: \"kubernetes.io/projected/8b5411ce-6a9f-4aaf-95d0-75050f1f6146-kube-api-access-8wk7w\") pod \"lws-controller-manager-78b4cf4c54-gljmf\" (UID: \"8b5411ce-6a9f-4aaf-95d0-75050f1f6146\") " pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.504770 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.504700 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b3e68d-303d-4a4e-81e4-c62ed13a66bf-tls-certs\") pod \"kube-auth-proxy-fc969b7d8-zrfbp\" (UID: \"82b3e68d-303d-4a4e-81e4-c62ed13a66bf\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" Jul 10 22:26:54.504770 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.504757 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b3e68d-303d-4a4e-81e4-c62ed13a66bf-tmp\") pod \"kube-auth-proxy-fc969b7d8-zrfbp\" (UID: \"82b3e68d-303d-4a4e-81e4-c62ed13a66bf\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" Jul 10 22:26:54.504911 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.504807 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg5qp\" (UniqueName: \"kubernetes.io/projected/82b3e68d-303d-4a4e-81e4-c62ed13a66bf-kube-api-access-kg5qp\") pod \"kube-auth-proxy-fc969b7d8-zrfbp\" (UID: \"82b3e68d-303d-4a4e-81e4-c62ed13a66bf\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" Jul 10 22:26:54.506964 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.506933 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82b3e68d-303d-4a4e-81e4-c62ed13a66bf-tmp\") pod \"kube-auth-proxy-fc969b7d8-zrfbp\" (UID: \"82b3e68d-303d-4a4e-81e4-c62ed13a66bf\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" Jul 10 22:26:54.506964 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.506951 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82b3e68d-303d-4a4e-81e4-c62ed13a66bf-tls-certs\") pod \"kube-auth-proxy-fc969b7d8-zrfbp\" (UID: \"82b3e68d-303d-4a4e-81e4-c62ed13a66bf\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" Jul 10 22:26:54.527372 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.527353 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg5qp\" (UniqueName: \"kubernetes.io/projected/82b3e68d-303d-4a4e-81e4-c62ed13a66bf-kube-api-access-kg5qp\") pod \"kube-auth-proxy-fc969b7d8-zrfbp\" (UID: \"82b3e68d-303d-4a4e-81e4-c62ed13a66bf\") " pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" Jul 10 22:26:54.548244 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.548222 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:54.658781 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.658747 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" Jul 10 22:26:54.691994 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.691842 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf"] Jul 10 22:26:54.694283 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:26:54.694257 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5411ce_6a9f_4aaf_95d0_75050f1f6146.slice/crio-26e844138a4e3de69a0deadab6e4ae1c2f866020b45541f79f0c8346185ed431 WatchSource:0}: Error finding container 26e844138a4e3de69a0deadab6e4ae1c2f866020b45541f79f0c8346185ed431: Status 404 returned error can't find the container with id 26e844138a4e3de69a0deadab6e4ae1c2f866020b45541f79f0c8346185ed431 Jul 10 22:26:54.696538 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.696514 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:26:54.796240 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:54.796216 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp"] Jul 10 22:26:54.798966 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:26:54.798940 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b3e68d_303d_4a4e_81e4_c62ed13a66bf.slice/crio-7452861860a802e4824284ab88022c1245d12f7072c968f5bcc11549925991f7 WatchSource:0}: Error finding container 7452861860a802e4824284ab88022c1245d12f7072c968f5bcc11549925991f7: Status 404 returned error can't find the container with id 7452861860a802e4824284ab88022c1245d12f7072c968f5bcc11549925991f7 Jul 10 22:26:55.084271 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:55.084182 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" event={"ID":"82b3e68d-303d-4a4e-81e4-c62ed13a66bf","Type":"ContainerStarted","Data":"7452861860a802e4824284ab88022c1245d12f7072c968f5bcc11549925991f7"} Jul 10 22:26:55.085250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:55.085227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" event={"ID":"8b5411ce-6a9f-4aaf-95d0-75050f1f6146","Type":"ContainerStarted","Data":"26e844138a4e3de69a0deadab6e4ae1c2f866020b45541f79f0c8346185ed431"} Jul 10 22:26:58.103278 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:58.103232 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" event={"ID":"82b3e68d-303d-4a4e-81e4-c62ed13a66bf","Type":"ContainerStarted","Data":"f6951c8b4ab5f32885781d6cef8eef990ce20c6ed420d948c42bc849c89b42b7"} Jul 10 22:26:58.105030 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:58.105002 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" event={"ID":"8b5411ce-6a9f-4aaf-95d0-75050f1f6146","Type":"ContainerStarted","Data":"8a274e67037146cfeb85b390484de7ea716e3f41b29f9d7d2067514f4a083562"} Jul 10 22:26:58.105146 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:58.105126 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:26:58.121129 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:58.121083 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-fc969b7d8-zrfbp" podStartSLOduration=0.946437525 podStartE2EDuration="4.12106818s" totalImagesPullingTime="3.174630655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:26:58.120469724 +0000 UTC m=+305.968751600" watchObservedRunningTime="2026-07-10 22:26:58.12106818 +0000 UTC m=+305.969350052" Jul 10 22:26:58.135542 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:26:58.135499 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" podStartSLOduration=0.859577341 podStartE2EDuration="4.135482932s" totalImagesPullingTime="3.275905591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:26:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:26:58.134027384 +0000 UTC m=+305.982309259" watchObservedRunningTime="2026-07-10 22:26:58.135482932 +0000 UTC m=+305.983764806" Jul 10 22:27:04.679041 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:04.679006 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc"] Jul 10 22:27:04.693699 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:04.693668 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" Jul 10 22:27:04.694495 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:04.694464 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc"] Jul 10 22:27:04.696832 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:04.696808 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-sk2vd\"" Jul 10 22:27:04.697170 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:04.697130 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 10 22:27:04.799274 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:04.799242 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e458ca23-f202-42c3-9db6-ffab0a97f27a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-29wlc\" (UID: \"e458ca23-f202-42c3-9db6-ffab0a97f27a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" Jul 10 22:27:04.799485 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:04.799292 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnv9\" (UniqueName: \"kubernetes.io/projected/e458ca23-f202-42c3-9db6-ffab0a97f27a-kube-api-access-4dnv9\") pod \"llmisvc-controller-manager-777c6bd7d5-29wlc\" (UID: \"e458ca23-f202-42c3-9db6-ffab0a97f27a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" Jul 10 22:27:04.900085 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:04.900051 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e458ca23-f202-42c3-9db6-ffab0a97f27a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-29wlc\" (UID: \"e458ca23-f202-42c3-9db6-ffab0a97f27a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" Jul 10 22:27:04.900268 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:04.900096 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnv9\" (UniqueName: \"kubernetes.io/projected/e458ca23-f202-42c3-9db6-ffab0a97f27a-kube-api-access-4dnv9\") pod \"llmisvc-controller-manager-777c6bd7d5-29wlc\" (UID: \"e458ca23-f202-42c3-9db6-ffab0a97f27a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" Jul 10 22:27:04.902555 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:04.902529 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e458ca23-f202-42c3-9db6-ffab0a97f27a-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-29wlc\" (UID: \"e458ca23-f202-42c3-9db6-ffab0a97f27a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" Jul 10 22:27:04.926682 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:04.926657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnv9\" (UniqueName: \"kubernetes.io/projected/e458ca23-f202-42c3-9db6-ffab0a97f27a-kube-api-access-4dnv9\") pod \"llmisvc-controller-manager-777c6bd7d5-29wlc\" (UID: \"e458ca23-f202-42c3-9db6-ffab0a97f27a\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" Jul 10 22:27:05.008466 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:05.008369 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" Jul 10 22:27:05.371794 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:05.371771 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc"] Jul 10 22:27:05.374343 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:27:05.374318 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode458ca23_f202_42c3_9db6_ffab0a97f27a.slice/crio-9b5766b843a41169801dd6242a225916a988d80bafdd53dc87ad9365521bb26f WatchSource:0}: Error finding container 9b5766b843a41169801dd6242a225916a988d80bafdd53dc87ad9365521bb26f: Status 404 returned error can't find the container with id 9b5766b843a41169801dd6242a225916a988d80bafdd53dc87ad9365521bb26f Jul 10 22:27:06.146797 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:06.146761 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" event={"ID":"e458ca23-f202-42c3-9db6-ffab0a97f27a","Type":"ContainerStarted","Data":"9b5766b843a41169801dd6242a225916a988d80bafdd53dc87ad9365521bb26f"} Jul 10 22:27:09.112886 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:09.112854 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-78b4cf4c54-gljmf" Jul 10 22:27:09.169406 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:09.169364 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" event={"ID":"e458ca23-f202-42c3-9db6-ffab0a97f27a","Type":"ContainerStarted","Data":"ee0d50c4e4bf2bdddcbf90a1dbbf8fc5c0929fb62504f891306a13feaf47783d"} Jul 10 22:27:09.169579 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:09.169489 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" Jul 10 22:27:09.184564 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:09.184515 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" podStartSLOduration=2.231867991 podStartE2EDuration="5.184500611s" totalImagesPullingTime="2.95263262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:27:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:27:09.183500241 +0000 UTC m=+317.031782117" watchObservedRunningTime="2026-07-10 22:27:09.184500611 +0000 UTC m=+317.032782485" Jul 10 22:27:40.176091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:27:40.176060 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-29wlc" Jul 10 22:28:40.066852 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.066815 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw"] Jul 10 22:28:40.069085 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.069069 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:40.073085 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.073063 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-c7znz\"" Jul 10 22:28:40.073293 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.073279 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 22:28:40.073465 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.073452 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 22:28:40.082050 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.082027 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw"] Jul 10 22:28:40.112879 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.112851 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d531564e-61cd-4ce6-8234-f3d1aa98d71b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-p76pw\" (UID: \"d531564e-61cd-4ce6-8234-f3d1aa98d71b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:40.112989 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.112882 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb4vj\" (UniqueName: \"kubernetes.io/projected/d531564e-61cd-4ce6-8234-f3d1aa98d71b-kube-api-access-rb4vj\") pod \"kuadrant-operator-controller-manager-55467d9f7-p76pw\" (UID: \"d531564e-61cd-4ce6-8234-f3d1aa98d71b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:40.206259 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.206229 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw"] Jul 10 22:28:40.206547 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:28:40.206526 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-rb4vj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" podUID="d531564e-61cd-4ce6-8234-f3d1aa98d71b" Jul 10 22:28:40.212873 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.212847 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw"] Jul 10 22:28:40.214309 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.214285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d531564e-61cd-4ce6-8234-f3d1aa98d71b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-p76pw\" (UID: \"d531564e-61cd-4ce6-8234-f3d1aa98d71b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:40.214428 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.214322 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb4vj\" (UniqueName: \"kubernetes.io/projected/d531564e-61cd-4ce6-8234-f3d1aa98d71b-kube-api-access-rb4vj\") pod \"kuadrant-operator-controller-manager-55467d9f7-p76pw\" (UID: \"d531564e-61cd-4ce6-8234-f3d1aa98d71b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:40.214713 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.214690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d531564e-61cd-4ce6-8234-f3d1aa98d71b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-p76pw\" (UID: \"d531564e-61cd-4ce6-8234-f3d1aa98d71b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:40.216596 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:28:40.216574 2559 projected.go:196] Error preparing data for projected volume kube-api-access-rb4vj for pod kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw: failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-140-116.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-140-116.ec2.internal' and this object Jul 10 22:28:40.216680 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:28:40.216663 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d531564e-61cd-4ce6-8234-f3d1aa98d71b-kube-api-access-rb4vj podName:d531564e-61cd-4ce6-8234-f3d1aa98d71b nodeName:}" failed. No retries permitted until 2026-07-10 22:28:40.716640458 +0000 UTC m=+408.564922324 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rb4vj" (UniqueName: "kubernetes.io/projected/d531564e-61cd-4ce6-8234-f3d1aa98d71b-kube-api-access-rb4vj") pod "kuadrant-operator-controller-manager-55467d9f7-p76pw" (UID: "d531564e-61cd-4ce6-8234-f3d1aa98d71b") : failed to fetch token: serviceaccounts "kuadrant-operator-controller-manager" is forbidden: User "system:node:ip-10-0-140-116.ec2.internal" cannot create resource "serviceaccounts/token" in API group "" in the namespace "kuadrant-system": no relationship found between node 'ip-10-0-140-116.ec2.internal' and this object Jul 10 22:28:40.233651 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.233598 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f"] Jul 10 22:28:40.235966 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.235947 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:40.256692 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.256670 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f"] Jul 10 22:28:40.314861 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.314829 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8916e08f-8936-4919-b098-f4625cccc679-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-fjw8f\" (UID: \"8916e08f-8936-4919-b098-f4625cccc679\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:40.315008 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.314928 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4n8k\" (UniqueName: \"kubernetes.io/projected/8916e08f-8936-4919-b098-f4625cccc679-kube-api-access-q4n8k\") pod \"kuadrant-operator-controller-manager-55467d9f7-fjw8f\" (UID: \"8916e08f-8936-4919-b098-f4625cccc679\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:40.415961 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.415928 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4n8k\" (UniqueName: \"kubernetes.io/projected/8916e08f-8936-4919-b098-f4625cccc679-kube-api-access-q4n8k\") pod \"kuadrant-operator-controller-manager-55467d9f7-fjw8f\" (UID: \"8916e08f-8936-4919-b098-f4625cccc679\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:40.416115 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.416017 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8916e08f-8936-4919-b098-f4625cccc679-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-fjw8f\" (UID: \"8916e08f-8936-4919-b098-f4625cccc679\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:40.416373 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.416354 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8916e08f-8936-4919-b098-f4625cccc679-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-fjw8f\" (UID: \"8916e08f-8936-4919-b098-f4625cccc679\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:40.425433 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.425403 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4n8k\" (UniqueName: \"kubernetes.io/projected/8916e08f-8936-4919-b098-f4625cccc679-kube-api-access-q4n8k\") pod \"kuadrant-operator-controller-manager-55467d9f7-fjw8f\" (UID: \"8916e08f-8936-4919-b098-f4625cccc679\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:40.546400 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.546363 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:40.576299 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.576266 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:40.578413 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:28:40.578382 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-p76pw\" is forbidden: User \"system:node:ip-10-0-140-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-116.ec2.internal' and this object" podUID="d531564e-61cd-4ce6-8234-f3d1aa98d71b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:40.582627 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.582601 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:40.584745 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:28:40.584724 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-p76pw\" is forbidden: User \"system:node:ip-10-0-140-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-116.ec2.internal' and this object" podUID="d531564e-61cd-4ce6-8234-f3d1aa98d71b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:40.617243 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.617211 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d531564e-61cd-4ce6-8234-f3d1aa98d71b-extensions-socket-volume\") pod \"d531564e-61cd-4ce6-8234-f3d1aa98d71b\" (UID: \"d531564e-61cd-4ce6-8234-f3d1aa98d71b\") " Jul 10 22:28:40.617540 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.617523 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb4vj\" (UniqueName: \"kubernetes.io/projected/d531564e-61cd-4ce6-8234-f3d1aa98d71b-kube-api-access-rb4vj\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:28:40.617616 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.617529 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d531564e-61cd-4ce6-8234-f3d1aa98d71b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d531564e-61cd-4ce6-8234-f3d1aa98d71b" (UID: "d531564e-61cd-4ce6-8234-f3d1aa98d71b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:40.676748 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.676672 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f"] Jul 10 22:28:40.679912 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:28:40.679886 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8916e08f_8936_4919_b098_f4625cccc679.slice/crio-e52abdc8ba0de668e693ebe3c01d0fce460d5f45c961e98fc1bad4d772a4f1aa WatchSource:0}: Error finding container e52abdc8ba0de668e693ebe3c01d0fce460d5f45c961e98fc1bad4d772a4f1aa: Status 404 returned error can't find the container with id e52abdc8ba0de668e693ebe3c01d0fce460d5f45c961e98fc1bad4d772a4f1aa Jul 10 22:28:40.718778 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.718751 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d531564e-61cd-4ce6-8234-f3d1aa98d71b-extensions-socket-volume\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:28:40.725509 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:40.725479 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d531564e-61cd-4ce6-8234-f3d1aa98d71b" path="/var/lib/kubelet/pods/d531564e-61cd-4ce6-8234-f3d1aa98d71b/volumes" Jul 10 22:28:41.583407 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:41.583358 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" event={"ID":"8916e08f-8936-4919-b098-f4625cccc679","Type":"ContainerStarted","Data":"e52abdc8ba0de668e693ebe3c01d0fce460d5f45c961e98fc1bad4d772a4f1aa"} Jul 10 22:28:41.583783 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:41.583423 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:41.588157 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:28:41.588120 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-p76pw\" is forbidden: User \"system:node:ip-10-0-140-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-116.ec2.internal' and this object" podUID="d531564e-61cd-4ce6-8234-f3d1aa98d71b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:42.724052 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:28:42.723994 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-p76pw\" is forbidden: User \"system:node:ip-10-0-140-116.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-116.ec2.internal' and this object" podUID="d531564e-61cd-4ce6-8234-f3d1aa98d71b" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-p76pw" Jul 10 22:28:45.605788 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:45.605744 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" event={"ID":"8916e08f-8936-4919-b098-f4625cccc679","Type":"ContainerStarted","Data":"de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d"} Jul 10 22:28:45.606184 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:45.605854 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:45.625940 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:45.625893 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" podStartSLOduration=1.364556694 podStartE2EDuration="5.625878689s" totalImagesPullingTime="4.261321995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:28:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:28:45.623561808 +0000 UTC m=+413.471843684" watchObservedRunningTime="2026-07-10 22:28:45.625878689 +0000 UTC m=+413.474160563" Jul 10 22:28:56.334074 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.334040 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f"] Jul 10 22:28:56.334525 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.334292 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" podUID="8916e08f-8936-4919-b098-f4625cccc679" containerName="manager" containerID="cri-o://de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d" gracePeriod=10 Jul 10 22:28:56.336314 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.336288 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:56.578424 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.578401 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:56.657233 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.657195 2559 generic.go:356] "Generic (PLEG): container finished" podID="8916e08f-8936-4919-b098-f4625cccc679" containerID="de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d" exitCode=0 Jul 10 22:28:56.657423 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.657237 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" Jul 10 22:28:56.657423 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.657255 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" event={"ID":"8916e08f-8936-4919-b098-f4625cccc679","Type":"ContainerDied","Data":"de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d"} Jul 10 22:28:56.657423 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.657318 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f" event={"ID":"8916e08f-8936-4919-b098-f4625cccc679","Type":"ContainerDied","Data":"e52abdc8ba0de668e693ebe3c01d0fce460d5f45c961e98fc1bad4d772a4f1aa"} Jul 10 22:28:56.657423 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.657350 2559 scope.go:117] "RemoveContainer" containerID="de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d" Jul 10 22:28:56.663784 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.663763 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8916e08f-8936-4919-b098-f4625cccc679-extensions-socket-volume\") pod \"8916e08f-8936-4919-b098-f4625cccc679\" (UID: \"8916e08f-8936-4919-b098-f4625cccc679\") " Jul 10 22:28:56.663906 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.663803 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4n8k\" (UniqueName: \"kubernetes.io/projected/8916e08f-8936-4919-b098-f4625cccc679-kube-api-access-q4n8k\") pod \"8916e08f-8936-4919-b098-f4625cccc679\" (UID: \"8916e08f-8936-4919-b098-f4625cccc679\") " Jul 10 22:28:56.664113 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.664094 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8916e08f-8936-4919-b098-f4625cccc679-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8916e08f-8936-4919-b098-f4625cccc679" (UID: "8916e08f-8936-4919-b098-f4625cccc679"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:28:56.666258 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.666216 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8916e08f-8936-4919-b098-f4625cccc679-kube-api-access-q4n8k" (OuterVolumeSpecName: "kube-api-access-q4n8k") pod "8916e08f-8936-4919-b098-f4625cccc679" (UID: "8916e08f-8936-4919-b098-f4625cccc679"). InnerVolumeSpecName "kube-api-access-q4n8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:28:56.667499 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.667483 2559 scope.go:117] "RemoveContainer" containerID="de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d" Jul 10 22:28:56.667755 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:28:56.667729 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d\": container with ID starting with de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d not found: ID does not exist" containerID="de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d" Jul 10 22:28:56.667797 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.667759 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d"} err="failed to get container status \"de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d\": rpc error: code = NotFound desc = could not find container \"de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d\": container with ID starting with de1bdd5789cce09b98879380fb13045b6262c85a72c95a792e0965bd089f6d8d not found: ID does not exist" Jul 10 22:28:56.764985 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.764958 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8916e08f-8936-4919-b098-f4625cccc679-extensions-socket-volume\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:28:56.764985 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.764982 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4n8k\" (UniqueName: \"kubernetes.io/projected/8916e08f-8936-4919-b098-f4625cccc679-kube-api-access-q4n8k\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:28:56.972666 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.972594 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f"] Jul 10 22:28:56.976553 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:56.976526 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fjw8f"] Jul 10 22:28:58.725111 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:28:58.725077 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8916e08f-8936-4919-b098-f4625cccc679" path="/var/lib/kubelet/pods/8916e08f-8936-4919-b098-f4625cccc679/volumes" Jul 10 22:29:14.940791 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:14.940751 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:29:14.941366 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:14.941289 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8916e08f-8936-4919-b098-f4625cccc679" containerName="manager" Jul 10 22:29:14.941366 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:14.941308 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8916e08f-8936-4919-b098-f4625cccc679" containerName="manager" Jul 10 22:29:14.941525 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:14.941394 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8916e08f-8936-4919-b098-f4625cccc679" containerName="manager" Jul 10 22:29:14.944478 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:14.944459 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" Jul 10 22:29:14.946901 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:14.946877 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 22:29:14.947025 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:14.946877 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 10 22:29:14.947025 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:14.946910 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 22:29:14.947701 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:14.947683 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-tp8vl\"" Jul 10 22:29:14.954648 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:14.954628 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:29:14.984580 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:14.984552 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:29:15.122632 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.122603 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brqk5\" (UniqueName: \"kubernetes.io/projected/b1905375-7db8-4cf1-b41c-7d4d37c59c91-kube-api-access-brqk5\") pod \"limitador-limitador-54d6d988d5-lmrx2\" (UID: \"b1905375-7db8-4cf1-b41c-7d4d37c59c91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" Jul 10 22:29:15.122800 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.122659 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1905375-7db8-4cf1-b41c-7d4d37c59c91-config-file\") pod \"limitador-limitador-54d6d988d5-lmrx2\" (UID: \"b1905375-7db8-4cf1-b41c-7d4d37c59c91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" Jul 10 22:29:15.223941 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.223861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1905375-7db8-4cf1-b41c-7d4d37c59c91-config-file\") pod \"limitador-limitador-54d6d988d5-lmrx2\" (UID: \"b1905375-7db8-4cf1-b41c-7d4d37c59c91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" Jul 10 22:29:15.224086 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.223944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brqk5\" (UniqueName: \"kubernetes.io/projected/b1905375-7db8-4cf1-b41c-7d4d37c59c91-kube-api-access-brqk5\") pod \"limitador-limitador-54d6d988d5-lmrx2\" (UID: \"b1905375-7db8-4cf1-b41c-7d4d37c59c91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" Jul 10 22:29:15.224511 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.224492 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b1905375-7db8-4cf1-b41c-7d4d37c59c91-config-file\") pod \"limitador-limitador-54d6d988d5-lmrx2\" (UID: \"b1905375-7db8-4cf1-b41c-7d4d37c59c91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" Jul 10 22:29:15.232709 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.232688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brqk5\" (UniqueName: \"kubernetes.io/projected/b1905375-7db8-4cf1-b41c-7d4d37c59c91-kube-api-access-brqk5\") pod \"limitador-limitador-54d6d988d5-lmrx2\" (UID: \"b1905375-7db8-4cf1-b41c-7d4d37c59c91\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" Jul 10 22:29:15.255500 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.255475 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" Jul 10 22:29:15.385793 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.385670 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:29:15.388114 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:29:15.388089 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1905375_7db8_4cf1_b41c_7d4d37c59c91.slice/crio-ae2c82508973ec45e3ae84451a51bcefdd20c6162aab146cefbe5f7e5ed64593 WatchSource:0}: Error finding container ae2c82508973ec45e3ae84451a51bcefdd20c6162aab146cefbe5f7e5ed64593: Status 404 returned error can't find the container with id ae2c82508973ec45e3ae84451a51bcefdd20c6162aab146cefbe5f7e5ed64593 Jul 10 22:29:15.407135 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.407104 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-26zdm"] Jul 10 22:29:15.410390 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.410370 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-26zdm" Jul 10 22:29:15.412710 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.412691 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-jc8jd\"" Jul 10 22:29:15.418877 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.418858 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-26zdm"] Jul 10 22:29:15.526408 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.526339 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qczmx\" (UniqueName: \"kubernetes.io/projected/73115802-3f7e-494f-9afc-9b07a204a279-kube-api-access-qczmx\") pod \"authorino-58cd955675-26zdm\" (UID: \"73115802-3f7e-494f-9afc-9b07a204a279\") " pod="kuadrant-system/authorino-58cd955675-26zdm" Jul 10 22:29:15.627019 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.626993 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qczmx\" (UniqueName: \"kubernetes.io/projected/73115802-3f7e-494f-9afc-9b07a204a279-kube-api-access-qczmx\") pod \"authorino-58cd955675-26zdm\" (UID: \"73115802-3f7e-494f-9afc-9b07a204a279\") " pod="kuadrant-system/authorino-58cd955675-26zdm" Jul 10 22:29:15.637114 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.637079 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qczmx\" (UniqueName: \"kubernetes.io/projected/73115802-3f7e-494f-9afc-9b07a204a279-kube-api-access-qczmx\") pod \"authorino-58cd955675-26zdm\" (UID: \"73115802-3f7e-494f-9afc-9b07a204a279\") " pod="kuadrant-system/authorino-58cd955675-26zdm" Jul 10 22:29:15.665723 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.665698 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hxdx"] Jul 10 22:29:15.668330 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.668313 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9hxdx" Jul 10 22:29:15.676417 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.676393 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hxdx"] Jul 10 22:29:15.730880 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.730850 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-26zdm" Jul 10 22:29:15.752794 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.752751 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" event={"ID":"b1905375-7db8-4cf1-b41c-7d4d37c59c91","Type":"ContainerStarted","Data":"ae2c82508973ec45e3ae84451a51bcefdd20c6162aab146cefbe5f7e5ed64593"} Jul 10 22:29:15.829536 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.829491 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8lmj\" (UniqueName: \"kubernetes.io/projected/49ed811b-6b53-4d12-9df1-6d5a21bacca9-kube-api-access-f8lmj\") pod \"authorino-75d556bd77-9hxdx\" (UID: \"49ed811b-6b53-4d12-9df1-6d5a21bacca9\") " pod="kuadrant-system/authorino-75d556bd77-9hxdx" Jul 10 22:29:15.881340 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.881309 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-26zdm"] Jul 10 22:29:15.883515 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:29:15.883486 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73115802_3f7e_494f_9afc_9b07a204a279.slice/crio-25bc5cbee8927188618508ff54b630d597b2fdcb52182ee2ad3df8df9693cb63 WatchSource:0}: Error finding container 25bc5cbee8927188618508ff54b630d597b2fdcb52182ee2ad3df8df9693cb63: Status 404 returned error can't find the container with id 25bc5cbee8927188618508ff54b630d597b2fdcb52182ee2ad3df8df9693cb63 Jul 10 22:29:15.930656 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.930216 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8lmj\" (UniqueName: \"kubernetes.io/projected/49ed811b-6b53-4d12-9df1-6d5a21bacca9-kube-api-access-f8lmj\") pod \"authorino-75d556bd77-9hxdx\" (UID: \"49ed811b-6b53-4d12-9df1-6d5a21bacca9\") " pod="kuadrant-system/authorino-75d556bd77-9hxdx" Jul 10 22:29:15.946866 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.946830 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8lmj\" (UniqueName: \"kubernetes.io/projected/49ed811b-6b53-4d12-9df1-6d5a21bacca9-kube-api-access-f8lmj\") pod \"authorino-75d556bd77-9hxdx\" (UID: \"49ed811b-6b53-4d12-9df1-6d5a21bacca9\") " pod="kuadrant-system/authorino-75d556bd77-9hxdx" Jul 10 22:29:15.978189 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:15.978021 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9hxdx" Jul 10 22:29:16.140001 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:16.139944 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hxdx"] Jul 10 22:29:16.142595 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:29:16.142357 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49ed811b_6b53_4d12_9df1_6d5a21bacca9.slice/crio-4d5ae28daa77955ec2b179772722bb13747166310aa879d1a058c62599bb19ed WatchSource:0}: Error finding container 4d5ae28daa77955ec2b179772722bb13747166310aa879d1a058c62599bb19ed: Status 404 returned error can't find the container with id 4d5ae28daa77955ec2b179772722bb13747166310aa879d1a058c62599bb19ed Jul 10 22:29:16.761682 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:16.761608 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-26zdm" event={"ID":"73115802-3f7e-494f-9afc-9b07a204a279","Type":"ContainerStarted","Data":"25bc5cbee8927188618508ff54b630d597b2fdcb52182ee2ad3df8df9693cb63"} Jul 10 22:29:16.765020 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:16.764986 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9hxdx" event={"ID":"49ed811b-6b53-4d12-9df1-6d5a21bacca9","Type":"ContainerStarted","Data":"4d5ae28daa77955ec2b179772722bb13747166310aa879d1a058c62599bb19ed"} Jul 10 22:29:20.789786 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:20.789744 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" event={"ID":"b1905375-7db8-4cf1-b41c-7d4d37c59c91","Type":"ContainerStarted","Data":"1be8415de4641d5ce268f1ce9954efaba70bd9816c000e6d157851082841b100"} Jul 10 22:29:20.790272 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:20.789909 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" Jul 10 22:29:20.791266 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:20.791236 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-26zdm" event={"ID":"73115802-3f7e-494f-9afc-9b07a204a279","Type":"ContainerStarted","Data":"3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6"} Jul 10 22:29:20.792610 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:20.792580 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9hxdx" event={"ID":"49ed811b-6b53-4d12-9df1-6d5a21bacca9","Type":"ContainerStarted","Data":"d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117"} Jul 10 22:29:20.805978 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:20.805929 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" podStartSLOduration=1.8171682150000001 podStartE2EDuration="6.805914344s" totalImagesPullingTime="4.988746129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:29:20.805524269 +0000 UTC m=+448.653806139" watchObservedRunningTime="2026-07-10 22:29:20.805914344 +0000 UTC m=+448.654196220" Jul 10 22:29:20.827298 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:20.827253 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-26zdm" podStartSLOduration=1.346903668 podStartE2EDuration="5.827239452s" totalImagesPullingTime="4.480335784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:29:20.82568178 +0000 UTC m=+448.673963655" watchObservedRunningTime="2026-07-10 22:29:20.827239452 +0000 UTC m=+448.675521327" Jul 10 22:29:20.842969 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:20.842927 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-9hxdx" podStartSLOduration=1.650647148 podStartE2EDuration="5.84291592s" totalImagesPullingTime="4.192268772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:29:20.841282071 +0000 UTC m=+448.689563945" watchObservedRunningTime="2026-07-10 22:29:20.84291592 +0000 UTC m=+448.691197808" Jul 10 22:29:20.869640 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:20.869612 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-26zdm"] Jul 10 22:29:22.801474 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:22.801402 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-26zdm" podUID="73115802-3f7e-494f-9afc-9b07a204a279" containerName="authorino" containerID="cri-o://3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6" gracePeriod=30 Jul 10 22:29:23.035612 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.035586 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-26zdm" Jul 10 22:29:23.091628 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.091555 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qczmx\" (UniqueName: \"kubernetes.io/projected/73115802-3f7e-494f-9afc-9b07a204a279-kube-api-access-qczmx\") pod \"73115802-3f7e-494f-9afc-9b07a204a279\" (UID: \"73115802-3f7e-494f-9afc-9b07a204a279\") " Jul 10 22:29:23.093541 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.093515 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73115802-3f7e-494f-9afc-9b07a204a279-kube-api-access-qczmx" (OuterVolumeSpecName: "kube-api-access-qczmx") pod "73115802-3f7e-494f-9afc-9b07a204a279" (UID: "73115802-3f7e-494f-9afc-9b07a204a279"). InnerVolumeSpecName "kube-api-access-qczmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:29:23.192750 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.192723 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qczmx\" (UniqueName: \"kubernetes.io/projected/73115802-3f7e-494f-9afc-9b07a204a279-kube-api-access-qczmx\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:29:23.807360 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.807323 2559 generic.go:356] "Generic (PLEG): container finished" podID="73115802-3f7e-494f-9afc-9b07a204a279" containerID="3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6" exitCode=0 Jul 10 22:29:23.807360 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.807355 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-26zdm" Jul 10 22:29:23.807907 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.807407 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-26zdm" event={"ID":"73115802-3f7e-494f-9afc-9b07a204a279","Type":"ContainerDied","Data":"3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6"} Jul 10 22:29:23.807907 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.807471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-26zdm" event={"ID":"73115802-3f7e-494f-9afc-9b07a204a279","Type":"ContainerDied","Data":"25bc5cbee8927188618508ff54b630d597b2fdcb52182ee2ad3df8df9693cb63"} Jul 10 22:29:23.807907 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.807491 2559 scope.go:117] "RemoveContainer" containerID="3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6" Jul 10 22:29:23.816026 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.816008 2559 scope.go:117] "RemoveContainer" containerID="3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6" Jul 10 22:29:23.816269 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:29:23.816251 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6\": container with ID starting with 3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6 not found: ID does not exist" containerID="3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6" Jul 10 22:29:23.816320 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.816278 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6"} err="failed to get container status \"3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6\": rpc error: code = NotFound desc = could not find container \"3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6\": container with ID starting with 3d09ed70a63661c62737de56aaae7bb2eddac4cab254129615e41d189bdb88e6 not found: ID does not exist" Jul 10 22:29:23.828827 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.828806 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-26zdm"] Jul 10 22:29:23.834820 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:23.834798 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-26zdm"] Jul 10 22:29:24.725837 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:24.725807 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73115802-3f7e-494f-9afc-9b07a204a279" path="/var/lib/kubelet/pods/73115802-3f7e-494f-9afc-9b07a204a279/volumes" Jul 10 22:29:31.798754 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:31.798723 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lmrx2" Jul 10 22:29:49.269248 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.269216 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zgp8h"] Jul 10 22:29:49.269705 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.269622 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73115802-3f7e-494f-9afc-9b07a204a279" containerName="authorino" Jul 10 22:29:49.269705 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.269637 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="73115802-3f7e-494f-9afc-9b07a204a279" containerName="authorino" Jul 10 22:29:49.269705 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.269696 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="73115802-3f7e-494f-9afc-9b07a204a279" containerName="authorino" Jul 10 22:29:49.278674 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.278653 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zgp8h" Jul 10 22:29:49.283279 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.283258 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zgp8h"] Jul 10 22:29:49.411334 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.411299 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l242g\" (UniqueName: \"kubernetes.io/projected/2a2a8f27-52d7-44a4-ab2b-74341f4903d6-kube-api-access-l242g\") pod \"authorino-5d4b67c446-zgp8h\" (UID: \"2a2a8f27-52d7-44a4-ab2b-74341f4903d6\") " pod="kuadrant-system/authorino-5d4b67c446-zgp8h" Jul 10 22:29:49.462566 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.462541 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zgp8h"] Jul 10 22:29:49.462800 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:29:49.462783 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-l242g], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-zgp8h" podUID="2a2a8f27-52d7-44a4-ab2b-74341f4903d6" Jul 10 22:29:49.496723 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.496693 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-fltcn"] Jul 10 22:29:49.500298 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.500283 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-fltcn" Jul 10 22:29:49.502391 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.502371 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 10 22:29:49.508766 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.508745 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-fltcn"] Jul 10 22:29:49.512194 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.512173 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l242g\" (UniqueName: \"kubernetes.io/projected/2a2a8f27-52d7-44a4-ab2b-74341f4903d6-kube-api-access-l242g\") pod \"authorino-5d4b67c446-zgp8h\" (UID: \"2a2a8f27-52d7-44a4-ab2b-74341f4903d6\") " pod="kuadrant-system/authorino-5d4b67c446-zgp8h" Jul 10 22:29:49.522510 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.522432 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l242g\" (UniqueName: \"kubernetes.io/projected/2a2a8f27-52d7-44a4-ab2b-74341f4903d6-kube-api-access-l242g\") pod \"authorino-5d4b67c446-zgp8h\" (UID: \"2a2a8f27-52d7-44a4-ab2b-74341f4903d6\") " pod="kuadrant-system/authorino-5d4b67c446-zgp8h" Jul 10 22:29:49.546025 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.546000 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-fltcn"] Jul 10 22:29:49.546314 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:29:49.546290 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dxjxt tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-dxjxt tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-fltcn" podUID="a64a9fa2-24e7-4905-8fe1-86ab85fd19af" Jul 10 22:29:49.582282 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.582256 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5996fb74d7-8hcqh"] Jul 10 22:29:49.585898 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.585871 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5996fb74d7-8hcqh" Jul 10 22:29:49.595116 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.595089 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5996fb74d7-8hcqh"] Jul 10 22:29:49.613543 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.613510 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-tls-cert\") pod \"authorino-76877994-fltcn\" (UID: \"a64a9fa2-24e7-4905-8fe1-86ab85fd19af\") " pod="kuadrant-system/authorino-76877994-fltcn" Jul 10 22:29:49.613665 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.613552 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjxt\" (UniqueName: \"kubernetes.io/projected/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-kube-api-access-dxjxt\") pod \"authorino-76877994-fltcn\" (UID: \"a64a9fa2-24e7-4905-8fe1-86ab85fd19af\") " pod="kuadrant-system/authorino-76877994-fltcn" Jul 10 22:29:49.714872 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.714843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwrb4\" (UniqueName: \"kubernetes.io/projected/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-kube-api-access-bwrb4\") pod \"authorino-5996fb74d7-8hcqh\" (UID: \"3d8b7a7a-46f4-4296-bf86-3cf9ea023806\") " pod="kuadrant-system/authorino-5996fb74d7-8hcqh" Jul 10 22:29:49.715003 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.714887 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-tls-cert\") pod \"authorino-76877994-fltcn\" (UID: \"a64a9fa2-24e7-4905-8fe1-86ab85fd19af\") " pod="kuadrant-system/authorino-76877994-fltcn" Jul 10 22:29:49.715003 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.714909 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjxt\" (UniqueName: \"kubernetes.io/projected/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-kube-api-access-dxjxt\") pod \"authorino-76877994-fltcn\" (UID: \"a64a9fa2-24e7-4905-8fe1-86ab85fd19af\") " pod="kuadrant-system/authorino-76877994-fltcn" Jul 10 22:29:49.715003 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.714984 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-tls-cert\") pod \"authorino-5996fb74d7-8hcqh\" (UID: \"3d8b7a7a-46f4-4296-bf86-3cf9ea023806\") " pod="kuadrant-system/authorino-5996fb74d7-8hcqh" Jul 10 22:29:49.717287 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.717259 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-tls-cert\") pod \"authorino-76877994-fltcn\" (UID: \"a64a9fa2-24e7-4905-8fe1-86ab85fd19af\") " pod="kuadrant-system/authorino-76877994-fltcn" Jul 10 22:29:49.724593 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.724560 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjxt\" (UniqueName: \"kubernetes.io/projected/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-kube-api-access-dxjxt\") pod \"authorino-76877994-fltcn\" (UID: \"a64a9fa2-24e7-4905-8fe1-86ab85fd19af\") " pod="kuadrant-system/authorino-76877994-fltcn" Jul 10 22:29:49.816432 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.816359 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwrb4\" (UniqueName: \"kubernetes.io/projected/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-kube-api-access-bwrb4\") pod \"authorino-5996fb74d7-8hcqh\" (UID: \"3d8b7a7a-46f4-4296-bf86-3cf9ea023806\") " pod="kuadrant-system/authorino-5996fb74d7-8hcqh" Jul 10 22:29:49.816432 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.816433 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-tls-cert\") pod \"authorino-5996fb74d7-8hcqh\" (UID: \"3d8b7a7a-46f4-4296-bf86-3cf9ea023806\") " pod="kuadrant-system/authorino-5996fb74d7-8hcqh" Jul 10 22:29:49.818714 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.818693 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-tls-cert\") pod \"authorino-5996fb74d7-8hcqh\" (UID: \"3d8b7a7a-46f4-4296-bf86-3cf9ea023806\") " pod="kuadrant-system/authorino-5996fb74d7-8hcqh" Jul 10 22:29:49.824784 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.824760 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwrb4\" (UniqueName: \"kubernetes.io/projected/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-kube-api-access-bwrb4\") pod \"authorino-5996fb74d7-8hcqh\" (UID: \"3d8b7a7a-46f4-4296-bf86-3cf9ea023806\") " pod="kuadrant-system/authorino-5996fb74d7-8hcqh" Jul 10 22:29:49.895082 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.895059 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5996fb74d7-8hcqh" Jul 10 22:29:49.925680 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.925649 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zgp8h" Jul 10 22:29:49.925842 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.925649 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-fltcn" Jul 10 22:29:49.935931 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.935911 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zgp8h" Jul 10 22:29:49.938292 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:49.938251 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-fltcn" Jul 10 22:29:50.017669 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.017641 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxjxt\" (UniqueName: \"kubernetes.io/projected/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-kube-api-access-dxjxt\") pod \"a64a9fa2-24e7-4905-8fe1-86ab85fd19af\" (UID: \"a64a9fa2-24e7-4905-8fe1-86ab85fd19af\") " Jul 10 22:29:50.017795 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.017713 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l242g\" (UniqueName: \"kubernetes.io/projected/2a2a8f27-52d7-44a4-ab2b-74341f4903d6-kube-api-access-l242g\") pod \"2a2a8f27-52d7-44a4-ab2b-74341f4903d6\" (UID: \"2a2a8f27-52d7-44a4-ab2b-74341f4903d6\") " Jul 10 22:29:50.017795 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.017754 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-tls-cert\") pod \"a64a9fa2-24e7-4905-8fe1-86ab85fd19af\" (UID: \"a64a9fa2-24e7-4905-8fe1-86ab85fd19af\") " Jul 10 22:29:50.019699 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.019671 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-kube-api-access-dxjxt" (OuterVolumeSpecName: "kube-api-access-dxjxt") pod "a64a9fa2-24e7-4905-8fe1-86ab85fd19af" (UID: "a64a9fa2-24e7-4905-8fe1-86ab85fd19af"). InnerVolumeSpecName "kube-api-access-dxjxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:29:50.019803 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.019710 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a64a9fa2-24e7-4905-8fe1-86ab85fd19af" (UID: "a64a9fa2-24e7-4905-8fe1-86ab85fd19af"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:29:50.019803 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.019718 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2a8f27-52d7-44a4-ab2b-74341f4903d6-kube-api-access-l242g" (OuterVolumeSpecName: "kube-api-access-l242g") pod "2a2a8f27-52d7-44a4-ab2b-74341f4903d6" (UID: "2a2a8f27-52d7-44a4-ab2b-74341f4903d6"). InnerVolumeSpecName "kube-api-access-l242g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:29:50.027426 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.027340 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5996fb74d7-8hcqh"] Jul 10 22:29:50.030128 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:29:50.030087 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d8b7a7a_46f4_4296_bf86_3cf9ea023806.slice/crio-e2da9936030a55143e8c58d212543df7871bc3f1ff779fe84f826457bb510dcd WatchSource:0}: Error finding container e2da9936030a55143e8c58d212543df7871bc3f1ff779fe84f826457bb510dcd: Status 404 returned error can't find the container with id e2da9936030a55143e8c58d212543df7871bc3f1ff779fe84f826457bb510dcd Jul 10 22:29:50.119022 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.118992 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjxt\" (UniqueName: \"kubernetes.io/projected/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-kube-api-access-dxjxt\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:29:50.119022 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.119021 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l242g\" (UniqueName: \"kubernetes.io/projected/2a2a8f27-52d7-44a4-ab2b-74341f4903d6-kube-api-access-l242g\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:29:50.119200 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.119035 2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a64a9fa2-24e7-4905-8fe1-86ab85fd19af-tls-cert\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:29:50.931433 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.931403 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-fltcn" Jul 10 22:29:50.931433 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.931413 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5996fb74d7-8hcqh" event={"ID":"3d8b7a7a-46f4-4296-bf86-3cf9ea023806","Type":"ContainerStarted","Data":"44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab"} Jul 10 22:29:50.931897 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.931469 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5996fb74d7-8hcqh" event={"ID":"3d8b7a7a-46f4-4296-bf86-3cf9ea023806","Type":"ContainerStarted","Data":"e2da9936030a55143e8c58d212543df7871bc3f1ff779fe84f826457bb510dcd"} Jul 10 22:29:50.931897 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.931587 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-zgp8h" Jul 10 22:29:50.953017 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.952875 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5996fb74d7-8hcqh" podStartSLOduration=1.592446025 podStartE2EDuration="1.952858772s" totalImagesPullingTime="360.412747ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:29:50.95258277 +0000 UTC m=+478.800864644" watchObservedRunningTime="2026-07-10 22:29:50.952858772 +0000 UTC m=+478.801140645" Jul 10 22:29:50.979577 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.979546 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zgp8h"] Jul 10 22:29:50.986772 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.986696 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hxdx"] Jul 10 22:29:50.986941 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.986915 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-9hxdx" podUID="49ed811b-6b53-4d12-9df1-6d5a21bacca9" containerName="authorino" containerID="cri-o://d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117" gracePeriod=30 Jul 10 22:29:50.989171 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:50.989149 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-zgp8h"] Jul 10 22:29:51.008075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.008045 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-fltcn"] Jul 10 22:29:51.012476 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.012451 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-fltcn"] Jul 10 22:29:51.237136 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.237082 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9hxdx" Jul 10 22:29:51.329377 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.329351 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8lmj\" (UniqueName: \"kubernetes.io/projected/49ed811b-6b53-4d12-9df1-6d5a21bacca9-kube-api-access-f8lmj\") pod \"49ed811b-6b53-4d12-9df1-6d5a21bacca9\" (UID: \"49ed811b-6b53-4d12-9df1-6d5a21bacca9\") " Jul 10 22:29:51.331344 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.331322 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ed811b-6b53-4d12-9df1-6d5a21bacca9-kube-api-access-f8lmj" (OuterVolumeSpecName: "kube-api-access-f8lmj") pod "49ed811b-6b53-4d12-9df1-6d5a21bacca9" (UID: "49ed811b-6b53-4d12-9df1-6d5a21bacca9"). InnerVolumeSpecName "kube-api-access-f8lmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:29:51.430917 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.430888 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8lmj\" (UniqueName: \"kubernetes.io/projected/49ed811b-6b53-4d12-9df1-6d5a21bacca9-kube-api-access-f8lmj\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:29:51.937192 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.937160 2559 generic.go:356] "Generic (PLEG): container finished" podID="49ed811b-6b53-4d12-9df1-6d5a21bacca9" containerID="d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117" exitCode=0 Jul 10 22:29:51.937617 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.937198 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-9hxdx" Jul 10 22:29:51.937617 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.937245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9hxdx" event={"ID":"49ed811b-6b53-4d12-9df1-6d5a21bacca9","Type":"ContainerDied","Data":"d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117"} Jul 10 22:29:51.937617 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.937288 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-9hxdx" event={"ID":"49ed811b-6b53-4d12-9df1-6d5a21bacca9","Type":"ContainerDied","Data":"4d5ae28daa77955ec2b179772722bb13747166310aa879d1a058c62599bb19ed"} Jul 10 22:29:51.937617 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.937309 2559 scope.go:117] "RemoveContainer" containerID="d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117" Jul 10 22:29:51.946272 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.946258 2559 scope.go:117] "RemoveContainer" containerID="d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117" Jul 10 22:29:51.946539 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:29:51.946520 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117\": container with ID starting with d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117 not found: ID does not exist" containerID="d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117" Jul 10 22:29:51.946619 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.946547 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117"} err="failed to get container status \"d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117\": rpc error: code = NotFound desc = could not find container \"d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117\": container with ID starting with d985d427aeb7c5d1ff68024b91be4ca38612eeb9360a0dfa1b5c3148d6ae6117 not found: ID does not exist" Jul 10 22:29:51.957561 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.957541 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hxdx"] Jul 10 22:29:51.960772 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:51.960752 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-9hxdx"] Jul 10 22:29:52.725574 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:52.725543 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a2a8f27-52d7-44a4-ab2b-74341f4903d6" path="/var/lib/kubelet/pods/2a2a8f27-52d7-44a4-ab2b-74341f4903d6/volumes" Jul 10 22:29:52.725817 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:52.725797 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ed811b-6b53-4d12-9df1-6d5a21bacca9" path="/var/lib/kubelet/pods/49ed811b-6b53-4d12-9df1-6d5a21bacca9/volumes" Jul 10 22:29:52.726216 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:29:52.726198 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a64a9fa2-24e7-4905-8fe1-86ab85fd19af" path="/var/lib/kubelet/pods/a64a9fa2-24e7-4905-8fe1-86ab85fd19af/volumes" Jul 10 22:30:31.874453 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:30:31.874409 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:30:53.401880 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:30:53.401843 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:30:59.806886 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:30:59.806852 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:31:06.885574 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.885538 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt"] Jul 10 22:31:06.886217 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.886195 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ed811b-6b53-4d12-9df1-6d5a21bacca9" containerName="authorino" Jul 10 22:31:06.886265 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.886222 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ed811b-6b53-4d12-9df1-6d5a21bacca9" containerName="authorino" Jul 10 22:31:06.886342 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.886330 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ed811b-6b53-4d12-9df1-6d5a21bacca9" containerName="authorino" Jul 10 22:31:06.889989 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.889970 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:06.892296 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.892274 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 22:31:06.893221 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.893202 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 22:31:06.893338 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.893223 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 10 22:31:06.893403 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.893349 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-szs6p\"" Jul 10 22:31:06.898585 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.897984 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt"] Jul 10 22:31:06.975589 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.975561 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9p8z\" (UniqueName: \"kubernetes.io/projected/28bd1151-7fc0-424d-9487-0f6560cf8afc-kube-api-access-v9p8z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:06.975749 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.975606 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28bd1151-7fc0-424d-9487-0f6560cf8afc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:06.975749 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.975643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:06.975749 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.975702 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:06.975868 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.975754 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:06.975868 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:06.975798 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.077167 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.077131 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9p8z\" (UniqueName: \"kubernetes.io/projected/28bd1151-7fc0-424d-9487-0f6560cf8afc-kube-api-access-v9p8z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.077330 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.077177 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28bd1151-7fc0-424d-9487-0f6560cf8afc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.077330 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.077212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.077330 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.077232 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.077330 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.077255 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.077330 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.077272 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.077694 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.077668 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.077694 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.077691 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.077796 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.077731 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.079587 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.079561 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28bd1151-7fc0-424d-9487-0f6560cf8afc-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.079817 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.079800 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28bd1151-7fc0-424d-9487-0f6560cf8afc-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.085082 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.085063 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9p8z\" (UniqueName: \"kubernetes.io/projected/28bd1151-7fc0-424d-9487-0f6560cf8afc-kube-api-access-v9p8z\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zlrvt\" (UID: \"28bd1151-7fc0-424d-9487-0f6560cf8afc\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.202309 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.202224 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:07.334372 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:07.334345 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt"] Jul 10 22:31:07.336537 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:31:07.336508 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28bd1151_7fc0_424d_9487_0f6560cf8afc.slice/crio-54fb0c309ad461d1040551e70b652ae322005289aed07a5b3bac4ed19f40d101 WatchSource:0}: Error finding container 54fb0c309ad461d1040551e70b652ae322005289aed07a5b3bac4ed19f40d101: Status 404 returned error can't find the container with id 54fb0c309ad461d1040551e70b652ae322005289aed07a5b3bac4ed19f40d101 Jul 10 22:31:08.010410 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:08.010370 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:31:08.297933 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:08.297053 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" event={"ID":"28bd1151-7fc0-424d-9487-0f6560cf8afc","Type":"ContainerStarted","Data":"54fb0c309ad461d1040551e70b652ae322005289aed07a5b3bac4ed19f40d101"} Jul 10 22:31:09.303586 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:09.303551 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" event={"ID":"28bd1151-7fc0-424d-9487-0f6560cf8afc","Type":"ContainerStarted","Data":"97829b01177cb248a26a37aa3fbfbb3155166e8ec9bdc5bfb678973016ce9df6"} Jul 10 22:31:09.322552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:09.322498 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" podStartSLOduration=2.237185724 podStartE2EDuration="3.32248268s" totalImagesPullingTime="1.085296956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:31:09.320776858 +0000 UTC m=+557.169058734" watchObservedRunningTime="2026-07-10 22:31:09.32248268 +0000 UTC m=+557.170764556" Jul 10 22:31:17.203052 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:17.203008 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:17.203490 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:17.203080 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:17.215557 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:17.215533 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:17.351924 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:17.351893 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zlrvt" Jul 10 22:31:19.210863 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:19.210831 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:31:34.393325 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.393292 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg"] Jul 10 22:31:34.396878 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.396856 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.398881 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.398858 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 22:31:34.406719 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.406695 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg"] Jul 10 22:31:34.524570 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.524538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.524570 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.524573 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.524763 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.524595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhbhd\" (UniqueName: \"kubernetes.io/projected/37fb17be-c620-4345-aec6-3927248f7191-kube-api-access-vhbhd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.524763 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.524678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.524763 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.524734 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fb17be-c620-4345-aec6-3927248f7191-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.524862 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.524769 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.625772 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.625738 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fb17be-c620-4345-aec6-3927248f7191-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.625956 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.625785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.625956 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.625837 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.625956 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.625855 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.625956 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.625878 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhbhd\" (UniqueName: \"kubernetes.io/projected/37fb17be-c620-4345-aec6-3927248f7191-kube-api-access-vhbhd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.625956 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.625923 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.626480 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.626425 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.626480 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.626471 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.626647 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.626424 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.628149 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.628124 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37fb17be-c620-4345-aec6-3927248f7191-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.628314 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.628295 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37fb17be-c620-4345-aec6-3927248f7191-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.633115 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.633095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhbhd\" (UniqueName: \"kubernetes.io/projected/37fb17be-c620-4345-aec6-3927248f7191-kube-api-access-vhbhd\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-bznfg\" (UID: \"37fb17be-c620-4345-aec6-3927248f7191\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.707482 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.707394 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:34.837704 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:34.837678 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg"] Jul 10 22:31:34.839535 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:31:34.839502 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37fb17be_c620_4345_aec6_3927248f7191.slice/crio-ddf420897f1a2ba76a4c7a1c9ad5db8838c95593368e5eb55cce7ed4b39c36b2 WatchSource:0}: Error finding container ddf420897f1a2ba76a4c7a1c9ad5db8838c95593368e5eb55cce7ed4b39c36b2: Status 404 returned error can't find the container with id ddf420897f1a2ba76a4c7a1c9ad5db8838c95593368e5eb55cce7ed4b39c36b2 Jul 10 22:31:35.428799 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:35.428768 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" event={"ID":"37fb17be-c620-4345-aec6-3927248f7191","Type":"ContainerStarted","Data":"201da9dd46615a613729679304ee3cd3cd42b1661268e512790f7896a0c4e7db"} Jul 10 22:31:35.429247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:35.428831 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" event={"ID":"37fb17be-c620-4345-aec6-3927248f7191","Type":"ContainerStarted","Data":"ddf420897f1a2ba76a4c7a1c9ad5db8838c95593368e5eb55cce7ed4b39c36b2"} Jul 10 22:31:35.445908 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:35.445844 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" podStartSLOduration=1.228092729 podStartE2EDuration="1.445824456s" totalImagesPullingTime="217.731727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:31:35.444189156 +0000 UTC m=+583.292471030" watchObservedRunningTime="2026-07-10 22:31:35.445824456 +0000 UTC m=+583.294106333" Jul 10 22:31:36.507480 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:36.507433 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:31:41.779392 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.779361 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm"] Jul 10 22:31:41.785074 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.785050 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.787517 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.787491 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 10 22:31:41.793309 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.792934 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm"] Jul 10 22:31:41.893686 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.893650 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.893842 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.893696 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gdx4\" (UniqueName: \"kubernetes.io/projected/e35f2f47-9483-4212-a50b-d0f02b5f7862-kube-api-access-7gdx4\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.893842 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.893786 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e35f2f47-9483-4212-a50b-d0f02b5f7862-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.893842 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.893828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.893957 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.893912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.893957 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.893940 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.994968 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.994935 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.994968 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.994972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gdx4\" (UniqueName: \"kubernetes.io/projected/e35f2f47-9483-4212-a50b-d0f02b5f7862-kube-api-access-7gdx4\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.995216 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.995007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e35f2f47-9483-4212-a50b-d0f02b5f7862-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.995216 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.995117 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.995335 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.995220 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.995335 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.995242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.995465 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.995403 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.995637 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.995615 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.995679 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.995625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.997327 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.997305 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e35f2f47-9483-4212-a50b-d0f02b5f7862-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:41.997633 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:41.997616 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e35f2f47-9483-4212-a50b-d0f02b5f7862-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:42.003210 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:42.003188 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gdx4\" (UniqueName: \"kubernetes.io/projected/e35f2f47-9483-4212-a50b-d0f02b5f7862-kube-api-access-7gdx4\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm\" (UID: \"e35f2f47-9483-4212-a50b-d0f02b5f7862\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:42.095873 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:42.095793 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:42.226887 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:42.226862 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm"] Jul 10 22:31:42.229148 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:31:42.229123 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode35f2f47_9483_4212_a50b_d0f02b5f7862.slice/crio-a94f3ee2685568f36eea584c18f099ad38ceccb550d46a77b8ad4e294a64dece WatchSource:0}: Error finding container a94f3ee2685568f36eea584c18f099ad38ceccb550d46a77b8ad4e294a64dece: Status 404 returned error can't find the container with id a94f3ee2685568f36eea584c18f099ad38ceccb550d46a77b8ad4e294a64dece Jul 10 22:31:42.465147 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:42.465115 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" event={"ID":"e35f2f47-9483-4212-a50b-d0f02b5f7862","Type":"ContainerStarted","Data":"a94f3ee2685568f36eea584c18f099ad38ceccb550d46a77b8ad4e294a64dece"} Jul 10 22:31:43.471060 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:43.471019 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" event={"ID":"e35f2f47-9483-4212-a50b-d0f02b5f7862","Type":"ContainerStarted","Data":"7a06288460e15ff59dfcd6ef5aafcfbbae78c6dd3e2833d2e63399057d67446b"} Jul 10 22:31:43.488538 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:43.488490 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" podStartSLOduration=2.297651399 podStartE2EDuration="2.488475126s" totalImagesPullingTime="190.823727ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:31:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:31:43.486286684 +0000 UTC m=+591.334568560" watchObservedRunningTime="2026-07-10 22:31:43.488475126 +0000 UTC m=+591.336757001" Jul 10 22:31:44.707625 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:44.707592 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:44.707996 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:44.707671 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:44.725004 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:44.724981 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:45.104430 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:45.104403 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:31:45.499496 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:45.499402 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-bznfg" Jul 10 22:31:52.096635 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:52.096597 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:52.097046 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:52.096647 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:52.108972 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:52.108943 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:52.523727 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:52.523702 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm" Jul 10 22:31:52.622334 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:52.622308 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:31:52.627394 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:31:52.627372 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:32:20.950875 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:20.950799 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65dddc6b7d-fmkh8"] Jul 10 22:32:20.954712 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:20.954686 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65dddc6b7d-fmkh8" Jul 10 22:32:20.961908 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:20.961884 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65dddc6b7d-fmkh8"] Jul 10 22:32:21.051824 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:21.051792 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/753b926f-254e-48e4-8eea-c8ff3e36f4cc-tls-cert\") pod \"authorino-65dddc6b7d-fmkh8\" (UID: \"753b926f-254e-48e4-8eea-c8ff3e36f4cc\") " pod="kuadrant-system/authorino-65dddc6b7d-fmkh8" Jul 10 22:32:21.051973 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:21.051835 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sqxd\" (UniqueName: \"kubernetes.io/projected/753b926f-254e-48e4-8eea-c8ff3e36f4cc-kube-api-access-2sqxd\") pod \"authorino-65dddc6b7d-fmkh8\" (UID: \"753b926f-254e-48e4-8eea-c8ff3e36f4cc\") " pod="kuadrant-system/authorino-65dddc6b7d-fmkh8" Jul 10 22:32:21.153143 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:21.153115 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/753b926f-254e-48e4-8eea-c8ff3e36f4cc-tls-cert\") pod \"authorino-65dddc6b7d-fmkh8\" (UID: \"753b926f-254e-48e4-8eea-c8ff3e36f4cc\") " pod="kuadrant-system/authorino-65dddc6b7d-fmkh8" Jul 10 22:32:21.153292 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:21.153166 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sqxd\" (UniqueName: \"kubernetes.io/projected/753b926f-254e-48e4-8eea-c8ff3e36f4cc-kube-api-access-2sqxd\") pod \"authorino-65dddc6b7d-fmkh8\" (UID: \"753b926f-254e-48e4-8eea-c8ff3e36f4cc\") " pod="kuadrant-system/authorino-65dddc6b7d-fmkh8" Jul 10 22:32:21.155866 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:21.155831 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/753b926f-254e-48e4-8eea-c8ff3e36f4cc-tls-cert\") pod \"authorino-65dddc6b7d-fmkh8\" (UID: \"753b926f-254e-48e4-8eea-c8ff3e36f4cc\") " pod="kuadrant-system/authorino-65dddc6b7d-fmkh8" Jul 10 22:32:21.160520 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:21.160496 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sqxd\" (UniqueName: \"kubernetes.io/projected/753b926f-254e-48e4-8eea-c8ff3e36f4cc-kube-api-access-2sqxd\") pod \"authorino-65dddc6b7d-fmkh8\" (UID: \"753b926f-254e-48e4-8eea-c8ff3e36f4cc\") " pod="kuadrant-system/authorino-65dddc6b7d-fmkh8" Jul 10 22:32:21.267730 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:21.267660 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65dddc6b7d-fmkh8" Jul 10 22:32:21.398887 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:21.398841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65dddc6b7d-fmkh8"] Jul 10 22:32:21.401480 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:32:21.401410 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod753b926f_254e_48e4_8eea_c8ff3e36f4cc.slice/crio-e6638d5b5dbb3618e08b2168ad74db7a44a64d1705facf683396b38c5d35c7ad WatchSource:0}: Error finding container e6638d5b5dbb3618e08b2168ad74db7a44a64d1705facf683396b38c5d35c7ad: Status 404 returned error can't find the container with id e6638d5b5dbb3618e08b2168ad74db7a44a64d1705facf683396b38c5d35c7ad Jul 10 22:32:21.402811 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:21.402794 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 22:32:21.658279 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:21.658245 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65dddc6b7d-fmkh8" event={"ID":"753b926f-254e-48e4-8eea-c8ff3e36f4cc","Type":"ContainerStarted","Data":"e6638d5b5dbb3618e08b2168ad74db7a44a64d1705facf683396b38c5d35c7ad"} Jul 10 22:32:22.665317 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:22.665275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65dddc6b7d-fmkh8" event={"ID":"753b926f-254e-48e4-8eea-c8ff3e36f4cc","Type":"ContainerStarted","Data":"424a5e327d991e592077e027b6cbb9ddc3f52807b534cecdf6ca6452bb03da4c"} Jul 10 22:32:22.681821 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:22.681411 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65dddc6b7d-fmkh8" podStartSLOduration=2.310989112 podStartE2EDuration="2.681390572s" totalImagesPullingTime="370.40146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:32:22.67999718 +0000 UTC m=+630.528279055" watchObservedRunningTime="2026-07-10 22:32:22.681390572 +0000 UTC m=+630.529672448" Jul 10 22:32:22.743865 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:22.741791 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5996fb74d7-8hcqh"] Jul 10 22:32:22.743865 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:22.742031 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5996fb74d7-8hcqh" podUID="3d8b7a7a-46f4-4296-bf86-3cf9ea023806" containerName="authorino" containerID="cri-o://44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab" gracePeriod=30 Jul 10 22:32:23.001933 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.001908 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5996fb74d7-8hcqh" Jul 10 22:32:23.072060 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.072027 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bwrb4\" (UniqueName: \"kubernetes.io/projected/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-kube-api-access-bwrb4\") pod \"3d8b7a7a-46f4-4296-bf86-3cf9ea023806\" (UID: \"3d8b7a7a-46f4-4296-bf86-3cf9ea023806\") " Jul 10 22:32:23.072060 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.072061 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-tls-cert\") pod \"3d8b7a7a-46f4-4296-bf86-3cf9ea023806\" (UID: \"3d8b7a7a-46f4-4296-bf86-3cf9ea023806\") " Jul 10 22:32:23.074280 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.074237 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-kube-api-access-bwrb4" (OuterVolumeSpecName: "kube-api-access-bwrb4") pod "3d8b7a7a-46f4-4296-bf86-3cf9ea023806" (UID: "3d8b7a7a-46f4-4296-bf86-3cf9ea023806"). InnerVolumeSpecName "kube-api-access-bwrb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:32:23.082291 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.082263 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "3d8b7a7a-46f4-4296-bf86-3cf9ea023806" (UID: "3d8b7a7a-46f4-4296-bf86-3cf9ea023806"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:32:23.173806 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.173775 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwrb4\" (UniqueName: \"kubernetes.io/projected/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-kube-api-access-bwrb4\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:32:23.173806 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.173805 2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3d8b7a7a-46f4-4296-bf86-3cf9ea023806-tls-cert\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:32:23.672471 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.672422 2559 generic.go:356] "Generic (PLEG): container finished" podID="3d8b7a7a-46f4-4296-bf86-3cf9ea023806" containerID="44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab" exitCode=0 Jul 10 22:32:23.672967 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.672487 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5996fb74d7-8hcqh" Jul 10 22:32:23.672967 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.672514 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5996fb74d7-8hcqh" event={"ID":"3d8b7a7a-46f4-4296-bf86-3cf9ea023806","Type":"ContainerDied","Data":"44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab"} Jul 10 22:32:23.672967 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.672560 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5996fb74d7-8hcqh" event={"ID":"3d8b7a7a-46f4-4296-bf86-3cf9ea023806","Type":"ContainerDied","Data":"e2da9936030a55143e8c58d212543df7871bc3f1ff779fe84f826457bb510dcd"} Jul 10 22:32:23.672967 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.672582 2559 scope.go:117] "RemoveContainer" containerID="44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab" Jul 10 22:32:23.682041 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.682013 2559 scope.go:117] "RemoveContainer" containerID="44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab" Jul 10 22:32:23.682334 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:32:23.682313 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab\": container with ID starting with 44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab not found: ID does not exist" containerID="44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab" Jul 10 22:32:23.682402 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.682348 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab"} err="failed to get container status \"44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab\": rpc error: code = NotFound desc = could not find container \"44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab\": container with ID starting with 44a1d5a78cfaca9b9aec9ad136398fadcc2c6513ca634b9df0bb2a3b498016ab not found: ID does not exist" Jul 10 22:32:23.696969 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.696941 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5996fb74d7-8hcqh"] Jul 10 22:32:23.700373 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:23.700350 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5996fb74d7-8hcqh"] Jul 10 22:32:24.728328 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:24.728279 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d8b7a7a-46f4-4296-bf86-3cf9ea023806" path="/var/lib/kubelet/pods/3d8b7a7a-46f4-4296-bf86-3cf9ea023806/volumes" Jul 10 22:32:50.409942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:50.409906 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:32:59.606285 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:32:59.606246 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:33:10.711075 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:33:10.711044 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:33:20.313504 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:33:20.313466 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:33:30.306176 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:33:30.306142 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:33:58.401156 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:33:58.401113 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:34:09.908449 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:34:09.908409 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:34:43.505471 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:34:43.504365 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:34:53.302360 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:34:53.302327 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:35:02.606063 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:35:02.606031 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:35:13.403064 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:35:13.403028 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:35:23.405498 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:35:23.405400 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:35:39.407460 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:35:39.407409 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:35:46.012353 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:35:46.008593 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:35:53.502314 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:35:53.502276 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:36:16.310554 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:36:16.310516 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:36:25.709863 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:36:25.709829 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:36:49.051587 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:36:49.051552 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:36:52.658784 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:36:52.658754 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:36:52.660085 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:36:52.660063 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:36:58.552189 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:36:58.552156 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:37:14.403536 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:37:14.403500 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:37:25.611813 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:37:25.611783 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:37:35.510156 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:37:35.510114 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:37:45.404660 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:37:45.404628 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:37:54.910152 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:37:54.910115 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:39:40.407851 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:39:40.407811 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:39:51.406281 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:39:51.406246 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:40:01.602670 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:40:01.602628 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:40:12.614927 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:40:12.614890 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:40:24.401291 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:40:24.401258 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:40:33.607182 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:40:33.607149 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:40:42.807641 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:40:42.807604 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:40:53.704727 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:40:53.704687 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:41:04.011103 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:41:04.011071 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:41:14.202958 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:41:14.202924 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:41:24.308482 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:41:24.308367 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:41:49.309678 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:41:49.309646 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:41:52.695636 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:41:52.695604 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:41:52.696079 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:41:52.695876 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:41:57.418549 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:41:57.418514 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:42:06.706396 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:42:06.706365 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:42:14.903250 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:42:14.903214 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:42:25.907946 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:42:25.907909 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:42:41.814371 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:42:41.814331 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:42:51.409753 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:42:51.409716 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:43:00.705134 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:43:00.705102 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:43:08.312205 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:43:08.312172 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:43:19.510727 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:43:19.510685 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:43:29.509934 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:43:29.509899 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:43:42.910197 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:43:42.910162 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:43:52.110328 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:43:52.110294 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:44:06.807247 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:44:06.807212 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:44:17.206987 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:44:17.206911 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:44:30.102859 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:44:30.102819 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:44:39.003688 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:44:39.003652 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:44:51.699782 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:44:51.699747 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:45:03.609822 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:45:03.609782 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:45:17.105987 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:45:17.105944 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:45:29.812942 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:45:29.812905 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:45:43.000861 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:45:43.000828 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:45:51.611742 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:45:51.611696 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:46:52.725802 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:46:52.725773 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:46:52.727313 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:46:52.727281 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:51:52.761787 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:51:52.761762 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:51:52.763823 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:51:52.763803 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:54:04.405461 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:54:04.405400 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:54:13.205292 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:54:13.205258 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:54:26.510309 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:54:26.510273 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:54:34.603648 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:54:34.603614 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:55:06.205840 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:06.205806 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:55:36.596757 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.596724 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8"] Jul 10 22:55:36.597196 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.597106 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8b7a7a-46f4-4296-bf86-3cf9ea023806" containerName="authorino" Jul 10 22:55:36.597196 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.597116 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8b7a7a-46f4-4296-bf86-3cf9ea023806" containerName="authorino" Jul 10 22:55:36.597196 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.597181 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8b7a7a-46f4-4296-bf86-3cf9ea023806" containerName="authorino" Jul 10 22:55:36.600202 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.600175 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.603921 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.603893 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-727180d9\"/\"default-dockercfg-ql6pq\"" Jul 10 22:55:36.604165 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.604145 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-727180d9\"/\"e2e-lifecycle-model-727180d9-kserve-self-signed-certs\"" Jul 10 22:55:36.606786 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.604277 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-727180d9\"/\"openshift-service-ca.crt\"" Jul 10 22:55:36.606972 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.604414 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-727180d9\"/\"kube-root-ca.crt\"" Jul 10 22:55:36.612299 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.612275 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8"] Jul 10 22:55:36.710244 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.710211 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-home\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.710244 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.710245 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/281f1129-92a9-499e-840c-5a4b6b1dfccd-tls-certs\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.710453 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.710282 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-dshm\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.710453 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.710355 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-tmp-dir\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.710453 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.710412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5jmp\" (UniqueName: \"kubernetes.io/projected/281f1129-92a9-499e-840c-5a4b6b1dfccd-kube-api-access-v5jmp\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.710556 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.710480 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-model-cache\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.811501 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.811467 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-home\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.811675 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.811507 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/281f1129-92a9-499e-840c-5a4b6b1dfccd-tls-certs\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.811675 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.811551 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-dshm\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.811675 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.811584 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-tmp-dir\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.811675 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.811646 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5jmp\" (UniqueName: \"kubernetes.io/projected/281f1129-92a9-499e-840c-5a4b6b1dfccd-kube-api-access-v5jmp\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.811894 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.811690 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-model-cache\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.811951 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.811903 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-home\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.812010 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.811940 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-tmp-dir\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.812050 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.812013 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-model-cache\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.813734 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.813715 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-dshm\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.814011 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.813992 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/281f1129-92a9-499e-840c-5a4b6b1dfccd-tls-certs\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.819329 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.819306 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5jmp\" (UniqueName: \"kubernetes.io/projected/281f1129-92a9-499e-840c-5a4b6b1dfccd-kube-api-access-v5jmp\") pod \"e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:36.913406 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:36.913378 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:37.043715 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:37.043692 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8"] Jul 10 22:55:37.045941 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:55:37.045907 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod281f1129_92a9_499e_840c_5a4b6b1dfccd.slice/crio-1509dcec766e42167c4778dcd4eb9e406be41147b804898877ade5c8cb26651e WatchSource:0}: Error finding container 1509dcec766e42167c4778dcd4eb9e406be41147b804898877ade5c8cb26651e: Status 404 returned error can't find the container with id 1509dcec766e42167c4778dcd4eb9e406be41147b804898877ade5c8cb26651e Jul 10 22:55:37.266758 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:37.266679 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" event={"ID":"281f1129-92a9-499e-840c-5a4b6b1dfccd","Type":"ContainerStarted","Data":"4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2"} Jul 10 22:55:37.266758 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:37.266727 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" event={"ID":"281f1129-92a9-499e-840c-5a4b6b1dfccd","Type":"ContainerStarted","Data":"1509dcec766e42167c4778dcd4eb9e406be41147b804898877ade5c8cb26651e"} Jul 10 22:55:37.282547 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:37.282501 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" podStartSLOduration=1.282485054 podStartE2EDuration="1.282485054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:55:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:55:37.281853944 +0000 UTC m=+2025.130135820" watchObservedRunningTime="2026-07-10 22:55:37.282485054 +0000 UTC m=+2025.130766930" Jul 10 22:55:40.009405 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:40.009369 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:55:46.913691 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:46.913657 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:46.914167 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:46.913725 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:46.926050 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:46.926025 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:47.323008 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:47.322925 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:55:57.909412 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:55:57.909375 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:56:08.302135 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:08.302097 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:56:08.882223 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:08.882190 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8"] Jul 10 22:56:23.903533 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:23.903416 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" podUID="281f1129-92a9-499e-840c-5a4b6b1dfccd" containerName="main" containerID="cri-o://4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2" gracePeriod=45 Jul 10 22:56:24.154290 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.154221 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:56:24.252753 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.252710 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-dshm\") pod \"281f1129-92a9-499e-840c-5a4b6b1dfccd\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " Jul 10 22:56:24.252753 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.252756 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5jmp\" (UniqueName: \"kubernetes.io/projected/281f1129-92a9-499e-840c-5a4b6b1dfccd-kube-api-access-v5jmp\") pod \"281f1129-92a9-499e-840c-5a4b6b1dfccd\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " Jul 10 22:56:24.253007 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.252804 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/281f1129-92a9-499e-840c-5a4b6b1dfccd-tls-certs\") pod \"281f1129-92a9-499e-840c-5a4b6b1dfccd\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " Jul 10 22:56:24.253007 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.252865 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-home\") pod \"281f1129-92a9-499e-840c-5a4b6b1dfccd\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " Jul 10 22:56:24.253007 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.252916 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-model-cache\") pod \"281f1129-92a9-499e-840c-5a4b6b1dfccd\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " Jul 10 22:56:24.253007 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.252988 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-tmp-dir\") pod \"281f1129-92a9-499e-840c-5a4b6b1dfccd\" (UID: \"281f1129-92a9-499e-840c-5a4b6b1dfccd\") " Jul 10 22:56:24.253220 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.253186 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-home" (OuterVolumeSpecName: "home") pod "281f1129-92a9-499e-840c-5a4b6b1dfccd" (UID: "281f1129-92a9-499e-840c-5a4b6b1dfccd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:56:24.253270 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.253220 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-model-cache" (OuterVolumeSpecName: "model-cache") pod "281f1129-92a9-499e-840c-5a4b6b1dfccd" (UID: "281f1129-92a9-499e-840c-5a4b6b1dfccd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:56:24.253338 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.253313 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "281f1129-92a9-499e-840c-5a4b6b1dfccd" (UID: "281f1129-92a9-499e-840c-5a4b6b1dfccd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:56:24.253394 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.253375 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-home\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:56:24.253651 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.253402 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-model-cache\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:56:24.255060 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.255038 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/281f1129-92a9-499e-840c-5a4b6b1dfccd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "281f1129-92a9-499e-840c-5a4b6b1dfccd" (UID: "281f1129-92a9-499e-840c-5a4b6b1dfccd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:56:24.255142 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.255084 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-dshm" (OuterVolumeSpecName: "dshm") pod "281f1129-92a9-499e-840c-5a4b6b1dfccd" (UID: "281f1129-92a9-499e-840c-5a4b6b1dfccd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:56:24.255142 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.255130 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281f1129-92a9-499e-840c-5a4b6b1dfccd-kube-api-access-v5jmp" (OuterVolumeSpecName: "kube-api-access-v5jmp") pod "281f1129-92a9-499e-840c-5a4b6b1dfccd" (UID: "281f1129-92a9-499e-840c-5a4b6b1dfccd"). InnerVolumeSpecName "kube-api-access-v5jmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:56:24.354002 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.353954 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-dshm\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:56:24.354002 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.353998 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5jmp\" (UniqueName: \"kubernetes.io/projected/281f1129-92a9-499e-840c-5a4b6b1dfccd-kube-api-access-v5jmp\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:56:24.354002 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.354010 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/281f1129-92a9-499e-840c-5a4b6b1dfccd-tls-certs\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:56:24.354245 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.354019 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/281f1129-92a9-499e-840c-5a4b6b1dfccd-tmp-dir\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:56:24.489482 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.489379 2559 generic.go:356] "Generic (PLEG): container finished" podID="281f1129-92a9-499e-840c-5a4b6b1dfccd" containerID="4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2" exitCode=0 Jul 10 22:56:24.489482 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.489423 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" Jul 10 22:56:24.489678 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.489479 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" event={"ID":"281f1129-92a9-499e-840c-5a4b6b1dfccd","Type":"ContainerDied","Data":"4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2"} Jul 10 22:56:24.489678 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.489525 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8" event={"ID":"281f1129-92a9-499e-840c-5a4b6b1dfccd","Type":"ContainerDied","Data":"1509dcec766e42167c4778dcd4eb9e406be41147b804898877ade5c8cb26651e"} Jul 10 22:56:24.489678 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.489547 2559 scope.go:117] "RemoveContainer" containerID="4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2" Jul 10 22:56:24.498718 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.498698 2559 scope.go:117] "RemoveContainer" containerID="4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2" Jul 10 22:56:24.498978 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:56:24.498961 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2\": container with ID starting with 4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2 not found: ID does not exist" containerID="4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2" Jul 10 22:56:24.499022 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.498992 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2"} err="failed to get container status \"4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2\": rpc error: code = NotFound desc = could not find container \"4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2\": container with ID starting with 4b427e953ed273d9f0c237eb942ac5571cca71611412bb5b73f65e3a4526c5d2 not found: ID does not exist" Jul 10 22:56:24.510811 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.510783 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8"] Jul 10 22:56:24.512530 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.512511 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-727180d9/e2e-lifecycle-model-727180d9-kserve-7d68d7595-558n8"] Jul 10 22:56:24.728529 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:24.728501 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="281f1129-92a9-499e-840c-5a4b6b1dfccd" path="/var/lib/kubelet/pods/281f1129-92a9-499e-840c-5a4b6b1dfccd/volumes" Jul 10 22:56:48.810986 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:48.810952 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:56:52.791628 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:52.791598 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:56:52.795860 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:56:52.795838 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 22:57:00.403495 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:57:00.403453 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:57:10.112403 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:57:10.108228 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:57:57.100777 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:57:57.100738 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:58:07.808610 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:07.808573 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:58:09.786929 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.786895 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d"] Jul 10 22:58:09.787537 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.787507 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="281f1129-92a9-499e-840c-5a4b6b1dfccd" containerName="main" Jul 10 22:58:09.787537 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.787528 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="281f1129-92a9-499e-840c-5a4b6b1dfccd" containerName="main" Jul 10 22:58:09.787689 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.787625 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="281f1129-92a9-499e-840c-5a4b6b1dfccd" containerName="main" Jul 10 22:58:09.791056 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.791037 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.794368 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.794346 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c8c45c\"/\"openshift-service-ca.crt\"" Jul 10 22:58:09.794368 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.794358 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c8c45c\"/\"test-model-c8c45c-kserve-self-signed-certs\"" Jul 10 22:58:09.794552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.794360 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c8c45c\"/\"default-dockercfg-qnz7m\"" Jul 10 22:58:09.794552 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.794385 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c8c45c\"/\"kube-root-ca.crt\"" Jul 10 22:58:09.798414 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.798392 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d"] Jul 10 22:58:09.807887 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.807857 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef7fb73-99fc-4fed-8662-b37976647e63-tls-certs\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.808005 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.807899 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-tmp-dir\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.808005 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.807917 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-dshm\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.808123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.808058 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-home\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.808123 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.808094 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkkpt\" (UniqueName: \"kubernetes.io/projected/0ef7fb73-99fc-4fed-8662-b37976647e63-kube-api-access-rkkpt\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.808281 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.808240 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-model-cache\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.909508 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.909474 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-tmp-dir\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.909669 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.909528 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-dshm\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.909669 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.909601 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-home\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.909669 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.909630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkkpt\" (UniqueName: \"kubernetes.io/projected/0ef7fb73-99fc-4fed-8662-b37976647e63-kube-api-access-rkkpt\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.909841 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.909720 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-model-cache\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.909841 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.909783 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef7fb73-99fc-4fed-8662-b37976647e63-tls-certs\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.909969 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.909835 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-tmp-dir\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.909969 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.909958 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-home\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.910079 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.910040 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-model-cache\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.911721 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.911697 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-dshm\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.911994 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.911978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef7fb73-99fc-4fed-8662-b37976647e63-tls-certs\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:09.917691 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:09.917664 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkkpt\" (UniqueName: \"kubernetes.io/projected/0ef7fb73-99fc-4fed-8662-b37976647e63-kube-api-access-rkkpt\") pod \"test-model-c8c45c-kserve-68484677df-m4d7d\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:10.101068 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:10.101045 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:10.234892 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:10.234867 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d"] Jul 10 22:58:10.236973 ip-10-0-140-116 kubenswrapper[2559]: W0710 22:58:10.236936 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ef7fb73_99fc_4fed_8662_b37976647e63.slice/crio-e2e743a1747a2ae6f4bdd6eaab0652e46b4c889ac2449db9048f997d786f8eec WatchSource:0}: Error finding container e2e743a1747a2ae6f4bdd6eaab0652e46b4c889ac2449db9048f997d786f8eec: Status 404 returned error can't find the container with id e2e743a1747a2ae6f4bdd6eaab0652e46b4c889ac2449db9048f997d786f8eec Jul 10 22:58:10.994141 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:10.994105 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" event={"ID":"0ef7fb73-99fc-4fed-8662-b37976647e63","Type":"ContainerStarted","Data":"9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014"} Jul 10 22:58:10.994141 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:10.994152 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" event={"ID":"0ef7fb73-99fc-4fed-8662-b37976647e63","Type":"ContainerStarted","Data":"e2e743a1747a2ae6f4bdd6eaab0652e46b4c889ac2449db9048f997d786f8eec"} Jul 10 22:58:11.010255 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:11.010206 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" podStartSLOduration=2.010191541 podStartE2EDuration="2.010191541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 22:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 22:58:11.00892288 +0000 UTC m=+2178.857204755" watchObservedRunningTime="2026-07-10 22:58:11.010191541 +0000 UTC m=+2178.858473417" Jul 10 22:58:19.910452 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:19.910402 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:58:20.101544 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:20.101510 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:20.101544 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:20.101553 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:20.114506 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:20.114473 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:58:21.056595 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:58:21.056563 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:59:08.606549 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:08.606513 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:59:36.409339 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:36.409304 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d"] Jul 10 22:59:41.005849 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:41.005814 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 22:59:51.430028 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.429982 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" podUID="0ef7fb73-99fc-4fed-8662-b37976647e63" containerName="main" containerID="cri-o://9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014" gracePeriod=45 Jul 10 22:59:51.679489 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.679464 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:59:51.725757 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.725682 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkkpt\" (UniqueName: \"kubernetes.io/projected/0ef7fb73-99fc-4fed-8662-b37976647e63-kube-api-access-rkkpt\") pod \"0ef7fb73-99fc-4fed-8662-b37976647e63\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " Jul 10 22:59:51.725897 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.725762 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-dshm\") pod \"0ef7fb73-99fc-4fed-8662-b37976647e63\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " Jul 10 22:59:51.725897 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.725791 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef7fb73-99fc-4fed-8662-b37976647e63-tls-certs\") pod \"0ef7fb73-99fc-4fed-8662-b37976647e63\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " Jul 10 22:59:51.725897 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.725814 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-home\") pod \"0ef7fb73-99fc-4fed-8662-b37976647e63\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " Jul 10 22:59:51.725897 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.725843 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-model-cache\") pod \"0ef7fb73-99fc-4fed-8662-b37976647e63\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " Jul 10 22:59:51.726091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.725927 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-tmp-dir\") pod \"0ef7fb73-99fc-4fed-8662-b37976647e63\" (UID: \"0ef7fb73-99fc-4fed-8662-b37976647e63\") " Jul 10 22:59:51.726091 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.726069 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-home" (OuterVolumeSpecName: "home") pod "0ef7fb73-99fc-4fed-8662-b37976647e63" (UID: "0ef7fb73-99fc-4fed-8662-b37976647e63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:59:51.726229 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.726192 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-model-cache" (OuterVolumeSpecName: "model-cache") pod "0ef7fb73-99fc-4fed-8662-b37976647e63" (UID: "0ef7fb73-99fc-4fed-8662-b37976647e63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:59:51.726346 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.726270 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0ef7fb73-99fc-4fed-8662-b37976647e63" (UID: "0ef7fb73-99fc-4fed-8662-b37976647e63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:59:51.726346 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.726289 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-home\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:59:51.727924 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.727905 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef7fb73-99fc-4fed-8662-b37976647e63-kube-api-access-rkkpt" (OuterVolumeSpecName: "kube-api-access-rkkpt") pod "0ef7fb73-99fc-4fed-8662-b37976647e63" (UID: "0ef7fb73-99fc-4fed-8662-b37976647e63"). InnerVolumeSpecName "kube-api-access-rkkpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 22:59:51.728316 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.728294 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ef7fb73-99fc-4fed-8662-b37976647e63-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0ef7fb73-99fc-4fed-8662-b37976647e63" (UID: "0ef7fb73-99fc-4fed-8662-b37976647e63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 22:59:51.728388 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.728321 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-dshm" (OuterVolumeSpecName: "dshm") pod "0ef7fb73-99fc-4fed-8662-b37976647e63" (UID: "0ef7fb73-99fc-4fed-8662-b37976647e63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 22:59:51.827630 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.827586 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-dshm\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:59:51.827630 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.827624 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0ef7fb73-99fc-4fed-8662-b37976647e63-tls-certs\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:59:51.827630 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.827635 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-model-cache\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:59:51.827630 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.827643 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ef7fb73-99fc-4fed-8662-b37976647e63-tmp-dir\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:59:51.827902 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:51.827651 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkkpt\" (UniqueName: \"kubernetes.io/projected/0ef7fb73-99fc-4fed-8662-b37976647e63-kube-api-access-rkkpt\") on node \"ip-10-0-140-116.ec2.internal\" DevicePath \"\"" Jul 10 22:59:52.489561 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:52.489527 2559 generic.go:356] "Generic (PLEG): container finished" podID="0ef7fb73-99fc-4fed-8662-b37976647e63" containerID="9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014" exitCode=0 Jul 10 22:59:52.490005 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:52.489570 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" Jul 10 22:59:52.490005 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:52.489615 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" event={"ID":"0ef7fb73-99fc-4fed-8662-b37976647e63","Type":"ContainerDied","Data":"9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014"} Jul 10 22:59:52.490005 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:52.489651 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d" event={"ID":"0ef7fb73-99fc-4fed-8662-b37976647e63","Type":"ContainerDied","Data":"e2e743a1747a2ae6f4bdd6eaab0652e46b4c889ac2449db9048f997d786f8eec"} Jul 10 22:59:52.490005 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:52.489668 2559 scope.go:117] "RemoveContainer" containerID="9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014" Jul 10 22:59:52.499473 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:52.499457 2559 scope.go:117] "RemoveContainer" containerID="9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014" Jul 10 22:59:52.499708 ip-10-0-140-116 kubenswrapper[2559]: E0710 22:59:52.499692 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014\": container with ID starting with 9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014 not found: ID does not exist" containerID="9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014" Jul 10 22:59:52.499769 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:52.499715 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014"} err="failed to get container status \"9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014\": rpc error: code = NotFound desc = could not find container \"9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014\": container with ID starting with 9835994904bbfe4a14ca19e2d2ed5517152d97987dc5536b5f84fdbd9dc23014 not found: ID does not exist" Jul 10 22:59:52.510616 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:52.510595 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d"] Jul 10 22:59:52.514377 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:52.514356 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c8c45c/test-model-c8c45c-kserve-68484677df-m4d7d"] Jul 10 22:59:52.726349 ip-10-0-140-116 kubenswrapper[2559]: I0710 22:59:52.726316 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ef7fb73-99fc-4fed-8662-b37976647e63" path="/var/lib/kubelet/pods/0ef7fb73-99fc-4fed-8662-b37976647e63/volumes" Jul 10 23:00:24.011416 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:00:24.011381 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lmrx2"] Jul 10 23:01:52.821499 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:01:52.821371 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 23:01:52.826681 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:01:52.826657 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-tkx48_fec2b764-2853-4ff8-95ff-b0264fb8e16e/ovn-acl-logging/0.log" Jul 10 23:03:49.000780 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:03:49.000741 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65dddc6b7d-fmkh8_753b926f-254e-48e4-8eea-c8ff3e36f4cc/authorino/0.log" Jul 10 23:03:54.025338 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:03:54.025307 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-29wlc_e458ca23-f202-42c3-9db6-ffab0a97f27a/manager/0.log" Jul 10 23:03:54.671081 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:03:54.671054 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x_27f48dab-3d0c-4ea9-a416-bcc8c659a938/copy-manifests/0.log" Jul 10 23:03:54.677694 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:03:54.677673 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x_27f48dab-3d0c-4ea9-a416-bcc8c659a938/manager/0.log" Jul 10 23:03:56.596681 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:03:56.596654 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65dddc6b7d-fmkh8_753b926f-254e-48e4-8eea-c8ff3e36f4cc/authorino/0.log" Jul 10 23:03:57.312598 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:03:57.312575 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lmrx2_b1905375-7db8-4cf1-b41c-7d4d37c59c91/limitador/0.log" Jul 10 23:03:58.152058 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:03:58.152028 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fc969b7d8-zrfbp_82b3e68d-303d-4a4e-81e4-c62ed13a66bf/kube-auth-proxy/0.log" Jul 10 23:03:59.202846 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:03:59.202813 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zlrvt_28bd1151-7fc0-424d-9487-0f6560cf8afc/main/0.log" Jul 10 23:03:59.460182 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:03:59.460104 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-bznfg_37fb17be-c620-4345-aec6-3927248f7191/main/0.log" Jul 10 23:03:59.581278 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:03:59.581247 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-682gm_e35f2f47-9483-4212-a50b-d0f02b5f7862/main/0.log" Jul 10 23:04:06.198085 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:06.198047 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-krhr5_976133c2-7307-4130-a56d-56f9727c7ba6/global-pull-secret-syncer/0.log" Jul 10 23:04:06.328786 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:06.328754 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9t2tv_8e7057db-a92a-460c-9cbc-7f9815afa729/konnectivity-agent/0.log" Jul 10 23:04:06.433980 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:06.433945 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-116.ec2.internal_419dafcf2fcd506b2efc42482d8111c7/haproxy/0.log" Jul 10 23:04:10.858350 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:10.858317 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-65dddc6b7d-fmkh8_753b926f-254e-48e4-8eea-c8ff3e36f4cc/authorino/0.log" Jul 10 23:04:11.165355 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:11.165276 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lmrx2_b1905375-7db8-4cf1-b41c-7d4d37c59c91/limitador/0.log" Jul 10 23:04:13.262889 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.262840 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6d4999c9b9-mb5wx_c4edbd6a-9a1b-4bf3-8653-aaeaecc00dec/metrics-server/0.log" Jul 10 23:04:13.289449 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.289411 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-vn2nl_1d8d7197-6c1f-4feb-a515-fded7e132782/monitoring-plugin/0.log" Jul 10 23:04:13.482235 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.482212 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbwlt_02307dd9-985a-402a-af0d-be3ed719cda6/node-exporter/0.log" Jul 10 23:04:13.507008 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.506989 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbwlt_02307dd9-985a-402a-af0d-be3ed719cda6/kube-rbac-proxy/0.log" Jul 10 23:04:13.529059 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.529036 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vbwlt_02307dd9-985a-402a-af0d-be3ed719cda6/init-textfile/0.log" Jul 10 23:04:13.558642 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.558618 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-97znx_07c406dc-3ad9-4c9b-964d-639136a35225/kube-rbac-proxy-main/0.log" Jul 10 23:04:13.582328 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.582306 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-97znx_07c406dc-3ad9-4c9b-964d-639136a35225/kube-rbac-proxy-self/0.log" Jul 10 23:04:13.611095 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.611076 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-97znx_07c406dc-3ad9-4c9b-964d-639136a35225/openshift-state-metrics/0.log" Jul 10 23:04:13.674398 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.674371 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dac91be4-3c64-4c2b-9617-aec2bfd5c512/prometheus/0.log" Jul 10 23:04:13.742753 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.742682 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dac91be4-3c64-4c2b-9617-aec2bfd5c512/config-reloader/0.log" Jul 10 23:04:13.770394 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.770372 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dac91be4-3c64-4c2b-9617-aec2bfd5c512/thanos-sidecar/0.log" Jul 10 23:04:13.802929 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.802905 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dac91be4-3c64-4c2b-9617-aec2bfd5c512/kube-rbac-proxy-web/0.log" Jul 10 23:04:13.830205 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.830172 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dac91be4-3c64-4c2b-9617-aec2bfd5c512/kube-rbac-proxy/0.log" Jul 10 23:04:13.856980 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.856954 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dac91be4-3c64-4c2b-9617-aec2bfd5c512/kube-rbac-proxy-thanos/0.log" Jul 10 23:04:13.881636 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.881610 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_dac91be4-3c64-4c2b-9617-aec2bfd5c512/init-config-reloader/0.log" Jul 10 23:04:13.958527 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.958502 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-g7v62_ef7cec55-1888-4b4d-bc27-c45e37991d6e/prometheus-operator-admission-webhook/0.log" Jul 10 23:04:13.990824 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:13.990800 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f44675f6-c7lbg_0290b83d-bf15-4cc2-aac0-7daeaaf40227/telemeter-client/0.log" Jul 10 23:04:14.013854 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:14.013788 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f44675f6-c7lbg_0290b83d-bf15-4cc2-aac0-7daeaaf40227/reload/0.log" Jul 10 23:04:14.037579 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:14.037544 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-f44675f6-c7lbg_0290b83d-bf15-4cc2-aac0-7daeaaf40227/kube-rbac-proxy/0.log" Jul 10 23:04:14.068351 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:14.068332 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8dff754-cxzkd_0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff/thanos-query/0.log" Jul 10 23:04:14.091571 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:14.091551 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8dff754-cxzkd_0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff/kube-rbac-proxy-web/0.log" Jul 10 23:04:14.113508 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:14.113489 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8dff754-cxzkd_0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff/kube-rbac-proxy/0.log" Jul 10 23:04:14.137678 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:14.137661 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8dff754-cxzkd_0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff/prom-label-proxy/0.log" Jul 10 23:04:14.168619 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:14.168601 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8dff754-cxzkd_0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff/kube-rbac-proxy-rules/0.log" Jul 10 23:04:14.190697 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:14.190679 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86b8dff754-cxzkd_0476ccfb-d276-4cf0-bd9f-32fdc2ebc7ff/kube-rbac-proxy-metrics/0.log" Jul 10 23:04:16.948721 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:16.948690 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7"] Jul 10 23:04:16.949123 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:16.949109 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ef7fb73-99fc-4fed-8662-b37976647e63" containerName="main" Jul 10 23:04:16.949166 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:16.949125 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ef7fb73-99fc-4fed-8662-b37976647e63" containerName="main" Jul 10 23:04:16.949225 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:16.949214 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ef7fb73-99fc-4fed-8662-b37976647e63" containerName="main" Jul 10 23:04:16.952638 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:16.952620 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:16.954916 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:16.954899 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qk7b9\"/\"kube-root-ca.crt\"" Jul 10 23:04:16.955676 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:16.955662 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qk7b9\"/\"openshift-service-ca.crt\"" Jul 10 23:04:16.955747 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:16.955672 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qk7b9\"/\"default-dockercfg-n2xp8\"" Jul 10 23:04:16.959609 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:16.959517 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7"] Jul 10 23:04:17.077188 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.077151 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-sys\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.077368 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.077230 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-podres\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.077368 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.077269 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrfhx\" (UniqueName: \"kubernetes.io/projected/78987e28-c4ea-4ceb-9683-03d9e65030c9-kube-api-access-wrfhx\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.077368 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.077333 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-lib-modules\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.077368 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.077359 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-proc\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.178730 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.178695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-lib-modules\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.178730 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.178734 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-proc\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.178975 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.178762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-sys\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.178975 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.178798 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-podres\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.178975 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.178817 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrfhx\" (UniqueName: \"kubernetes.io/projected/78987e28-c4ea-4ceb-9683-03d9e65030c9-kube-api-access-wrfhx\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.178975 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.178850 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-proc\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.178975 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.178905 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-lib-modules\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.178975 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.178908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-sys\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.178975 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.178972 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/78987e28-c4ea-4ceb-9683-03d9e65030c9-podres\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.186995 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.186968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrfhx\" (UniqueName: \"kubernetes.io/projected/78987e28-c4ea-4ceb-9683-03d9e65030c9-kube-api-access-wrfhx\") pod \"perf-node-gather-daemonset-n7nz7\" (UID: \"78987e28-c4ea-4ceb-9683-03d9e65030c9\") " pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.262984 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.262910 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.397813 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.397785 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7"] Jul 10 23:04:17.400195 ip-10-0-140-116 kubenswrapper[2559]: W0710 23:04:17.400169 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod78987e28_c4ea_4ceb_9683_03d9e65030c9.slice/crio-56dc5b88e05a5947883f2f5fafbf717c1921c9d729870327735058826309e7c7 WatchSource:0}: Error finding container 56dc5b88e05a5947883f2f5fafbf717c1921c9d729870327735058826309e7c7: Status 404 returned error can't find the container with id 56dc5b88e05a5947883f2f5fafbf717c1921c9d729870327735058826309e7c7 Jul 10 23:04:17.732984 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.732946 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" event={"ID":"78987e28-c4ea-4ceb-9683-03d9e65030c9","Type":"ContainerStarted","Data":"1fe9a4fe1f5aa61c1cd7725446b3bb0a21efa31faf2bb65047d6f305031064d7"} Jul 10 23:04:17.733172 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.733003 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" event={"ID":"78987e28-c4ea-4ceb-9683-03d9e65030c9","Type":"ContainerStarted","Data":"56dc5b88e05a5947883f2f5fafbf717c1921c9d729870327735058826309e7c7"} Jul 10 23:04:17.733172 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.733032 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:17.750176 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:17.750124 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" podStartSLOduration=1.750109214 podStartE2EDuration="1.750109214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 23:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 23:04:17.748850413 +0000 UTC m=+2545.597132287" watchObservedRunningTime="2026-07-10 23:04:17.750109214 +0000 UTC m=+2545.598391089" Jul 10 23:04:18.506429 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:18.506399 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8s7sv_4fc1d3dd-e770-4942-b849-c31dd37c52fa/dns/0.log" Jul 10 23:04:18.527928 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:18.527900 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8s7sv_4fc1d3dd-e770-4942-b849-c31dd37c52fa/kube-rbac-proxy/0.log" Jul 10 23:04:18.654451 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:18.654421 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-m5kxh_2cb9e8fd-ed42-42c3-8174-34e2ac5a51d7/dns-node-resolver/0.log" Jul 10 23:04:19.228582 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:19.228554 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7hj6f_1d1f9ea3-e113-4617-bd09-fc1b793a2bbe/node-ca/0.log" Jul 10 23:04:20.333152 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:20.333120 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fc969b7d8-zrfbp_82b3e68d-303d-4a4e-81e4-c62ed13a66bf/kube-auth-proxy/0.log" Jul 10 23:04:21.218632 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:21.218603 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-s7t6h_8ad3159f-50bb-4fc7-af39-e85508a1575d/serve-healthcheck-canary/0.log" Jul 10 23:04:21.693750 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:21.693715 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-k8jhp_64cfd78b-f6a5-4da0-ab73-dd6928c55dae/insights-operator/0.log" Jul 10 23:04:21.694332 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:21.694315 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-k8jhp_64cfd78b-f6a5-4da0-ab73-dd6928c55dae/insights-operator/1.log" Jul 10 23:04:21.784328 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:21.784306 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kcnnj_fc1678f8-4523-4495-90d5-a35bd0fa292d/kube-rbac-proxy/0.log" Jul 10 23:04:21.805130 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:21.805110 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kcnnj_fc1678f8-4523-4495-90d5-a35bd0fa292d/exporter/0.log" Jul 10 23:04:21.828525 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:21.828500 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kcnnj_fc1678f8-4523-4495-90d5-a35bd0fa292d/extractor/0.log" Jul 10 23:04:23.747099 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:23.747072 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qk7b9/perf-node-gather-daemonset-n7nz7" Jul 10 23:04:24.122961 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:24.122927 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-29wlc_e458ca23-f202-42c3-9db6-ffab0a97f27a/manager/0.log" Jul 10 23:04:24.419958 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:24.419881 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x_27f48dab-3d0c-4ea9-a416-bcc8c659a938/manager/0.log" Jul 10 23:04:24.439151 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:24.439121 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-58f8cd5b6f-t5x8x_27f48dab-3d0c-4ea9-a416-bcc8c659a938/copy-manifests/0.log" Jul 10 23:04:25.693666 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:25.693636 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-78b4cf4c54-gljmf_8b5411ce-6a9f-4aaf-95d0-75050f1f6146/manager/0.log" Jul 10 23:04:26.987960 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:26.987927 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fc969b7d8-zrfbp_82b3e68d-303d-4a4e-81e4-c62ed13a66bf/kube-auth-proxy/0.log" Jul 10 23:04:28.417507 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:28.417481 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-fc969b7d8-zrfbp_82b3e68d-303d-4a4e-81e4-c62ed13a66bf/kube-auth-proxy/0.log" Jul 10 23:04:31.768844 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:31.768813 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-mtt7p_2be24f5f-4532-4c5f-bebf-5c5b76a699cd/kube-storage-version-migrator-operator/1.log" Jul 10 23:04:31.769658 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:31.769641 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-mtt7p_2be24f5f-4532-4c5f-bebf-5c5b76a699cd/kube-storage-version-migrator-operator/0.log" Jul 10 23:04:33.067658 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:33.067629 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfvl2_6171ceb6-38d3-46f3-a504-42094bd28cde/kube-multus-additional-cni-plugins/0.log" Jul 10 23:04:33.088665 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:33.088641 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfvl2_6171ceb6-38d3-46f3-a504-42094bd28cde/egress-router-binary-copy/0.log" Jul 10 23:04:33.108796 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:33.108774 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfvl2_6171ceb6-38d3-46f3-a504-42094bd28cde/cni-plugins/0.log" Jul 10 23:04:33.129773 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:33.129755 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfvl2_6171ceb6-38d3-46f3-a504-42094bd28cde/bond-cni-plugin/0.log" Jul 10 23:04:33.150057 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:33.150011 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfvl2_6171ceb6-38d3-46f3-a504-42094bd28cde/routeoverride-cni/0.log" Jul 10 23:04:33.170885 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:33.170869 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfvl2_6171ceb6-38d3-46f3-a504-42094bd28cde/whereabouts-cni-bincopy/0.log" Jul 10 23:04:33.190621 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:33.190601 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-nfvl2_6171ceb6-38d3-46f3-a504-42094bd28cde/whereabouts-cni/0.log" Jul 10 23:04:33.254297 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:33.254246 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fmj2g_7c6ad821-e27e-49a2-85ec-354c19403be5/kube-multus/0.log" Jul 10 23:04:33.387095 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:33.387070 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wbxsz_81ca9b9c-e95f-4ac1-9e97-96169222847b/network-metrics-daemon/0.log" Jul 10 23:04:33.408376 ip-10-0-140-116 kubenswrapper[2559]: I0710 23:04:33.408353 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-wbxsz_81ca9b9c-e95f-4ac1-9e97-96169222847b/kube-rbac-proxy/0.log"