Sep 16 18:43:34.538651 ip-10-0-134-125 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 16 18:43:34.538665 ip-10-0-134-125 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 16 18:43:34.538675 ip-10-0-134-125 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 16 18:43:34.538877 ip-10-0-134-125 systemd[1]: Failed to start Kubernetes Kubelet. Sep 16 18:43:44.587360 ip-10-0-134-125 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 16 18:43:44.587375 ip-10-0-134-125 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 88847a0766954375a8d0c79c62d16b33 -- Sep 16 18:45:39.557178 ip-10-0-134-125 systemd[1]: Starting Kubernetes Kubelet... Sep 16 18:45:39.953560 ip-10-0-134-125 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:39.953560 ip-10-0-134-125 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 18:45:39.953560 ip-10-0-134-125 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:39.953560 ip-10-0-134-125 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 18:45:39.953560 ip-10-0-134-125 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 18:45:39.954547 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.954452 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 18:45:39.957817 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957786 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:39.957817 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957816 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957824 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957851 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957856 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957860 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957865 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957869 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957874 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957879 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957883 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957888 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957899 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957904 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957909 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957918 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957922 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957926 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957931 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957935 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957940 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:39.957975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957944 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957949 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957953 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957960 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957967 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957972 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957982 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957987 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957992 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.957997 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958001 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958006 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958010 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958014 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958019 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958023 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958027 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958032 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958036 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:39.958716 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958048 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958052 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958056 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958061 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958066 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958070 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958075 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958079 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958084 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958088 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958092 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958096 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958105 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958110 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958114 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958120 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958124 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958129 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958134 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:39.959257 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958138 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958142 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958146 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958150 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958154 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958159 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958168 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958172 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958176 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958181 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958185 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958190 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958194 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958198 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958203 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958207 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958211 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958215 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958225 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958230 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:39.959735 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958237 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958242 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958246 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958250 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958257 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958263 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958268 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958272 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958281 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958292 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958298 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958303 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958307 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958312 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958316 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958321 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958324 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958329 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958333 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.958338 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:39.960288 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959605 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959617 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959622 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959625 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959627 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959630 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959634 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959636 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959639 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959642 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959645 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959648 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959651 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959655 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959657 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959660 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959663 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959666 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959669 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959672 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:39.960787 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959675 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959677 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959680 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959682 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959685 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959688 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959690 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959693 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959696 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959698 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959701 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959704 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959707 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959710 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959713 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959715 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959718 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959721 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959724 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959727 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:39.961311 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959729 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959734 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959738 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959741 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959746 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959749 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959752 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959755 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959758 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959761 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959764 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959767 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959770 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959772 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959775 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959777 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959781 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959783 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959786 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:39.961798 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959788 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959791 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959794 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959796 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959799 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959806 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959808 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959812 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959815 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959818 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959821 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959824 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959826 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959842 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959845 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959847 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959850 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959853 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959855 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959858 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:39.962274 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959861 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959863 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959866 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959868 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959873 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959875 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959878 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959880 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959883 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959885 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959888 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959891 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959893 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959896 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959898 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959900 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959905 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959911 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959914 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:39.962763 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.959917 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960360 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960366 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960369 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960372 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960375 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960378 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960381 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960383 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960386 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960389 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960392 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960395 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960399 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960402 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960405 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960408 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960411 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960415 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:39.963246 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960418 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960420 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960423 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960426 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960428 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960431 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960433 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960436 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960438 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960441 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960444 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960447 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960450 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960453 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960455 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960458 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960460 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960463 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960466 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960468 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:39.963728 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960471 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960475 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960478 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960481 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960484 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960486 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960489 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960492 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960494 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960496 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960499 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960502 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960505 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960507 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960510 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960512 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960515 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960517 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960520 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:39.964252 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960522 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960525 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960527 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960530 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960533 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960535 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960538 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960541 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960543 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960547 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960550 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960553 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960557 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960559 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960562 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960564 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960567 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960569 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960572 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960574 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:39.964748 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960577 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960579 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960582 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960584 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960586 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960591 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960593 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960596 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960599 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960602 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960604 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960606 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960609 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960611 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960614 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960617 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960620 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960622 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960625 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960628 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:39.965249 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960630 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.960633 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960715 2563 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960757 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960764 2563 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960769 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960773 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960777 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960782 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960787 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960790 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960793 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960797 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960801 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960804 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960807 2563 flags.go:64] FLAG: --cgroup-root="" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960810 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960814 2563 flags.go:64] FLAG: --client-ca-file="" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960817 2563 flags.go:64] FLAG: --cloud-provider="external" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960821 2563 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960825 2563 flags.go:64] FLAG: --cluster-domain="" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960828 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960845 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960848 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 18:45:39.965734 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960851 2563 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960856 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960859 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960863 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960867 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960870 2563 flags.go:64] FLAG: --contention-profiling="false" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960873 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960876 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960880 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960883 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960887 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960890 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960893 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960896 2563 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960899 2563 flags.go:64] FLAG: --enable-server="true" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960902 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960907 2563 flags.go:64] FLAG: --event-burst="100" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960910 2563 flags.go:64] FLAG: --event-qps="50" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960913 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960916 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960920 2563 flags.go:64] FLAG: --eviction-hard="" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960923 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960926 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960930 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960932 2563 flags.go:64] FLAG: --eviction-soft="" Sep 16 18:45:39.966330 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960935 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960938 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960942 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960945 2563 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960948 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960951 2563 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960954 2563 flags.go:64] FLAG: --feature-gates="" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960957 2563 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960961 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960964 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960969 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960972 2563 flags.go:64] FLAG: --healthz-port="10248" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960975 2563 flags.go:64] FLAG: --help="false" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960979 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-134-125.ec2.internal" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960982 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960985 2563 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960988 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960991 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960995 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.960998 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961001 2563 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961004 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961007 2563 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 18:45:39.966951 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961010 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961013 2563 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961016 2563 flags.go:64] FLAG: --kube-reserved="" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961019 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961022 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961025 2563 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961028 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961031 2563 flags.go:64] FLAG: --lock-file="" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961034 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961037 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961040 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961045 2563 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961049 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961052 2563 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961055 2563 flags.go:64] FLAG: --logging-format="text" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961058 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961061 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961064 2563 flags.go:64] FLAG: --manifest-url="" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961067 2563 flags.go:64] FLAG: --manifest-url-header="" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961075 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961079 2563 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961083 2563 flags.go:64] FLAG: --max-pods="110" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961086 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961089 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961092 2563 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 18:45:39.967567 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961095 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961098 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961101 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961104 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961113 2563 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961116 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961119 2563 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961122 2563 flags.go:64] FLAG: --pod-cidr="" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961125 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961131 2563 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961134 2563 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961137 2563 flags.go:64] FLAG: --pods-per-core="0" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961140 2563 flags.go:64] FLAG: --port="10250" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961143 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961146 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-027548d683591ab66" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961150 2563 flags.go:64] FLAG: --qos-reserved="" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961153 2563 flags.go:64] FLAG: --read-only-port="10255" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961156 2563 flags.go:64] FLAG: --register-node="true" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961159 2563 flags.go:64] FLAG: --register-with-taints="" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961163 2563 flags.go:64] FLAG: --registry-burst="10" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961167 2563 flags.go:64] FLAG: --registry-qps="5" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961170 2563 flags.go:64] FLAG: --reserved-cpus="" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961173 2563 flags.go:64] FLAG: --reserved-memory="" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961176 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961179 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 18:45:39.968240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961182 2563 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961186 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961191 2563 flags.go:64] FLAG: --runonce="false" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961194 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961197 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961200 2563 flags.go:64] FLAG: --seccomp-default="false" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961203 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961206 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961209 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961212 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961215 2563 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961218 2563 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961221 2563 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961224 2563 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961227 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961230 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961233 2563 flags.go:64] FLAG: --system-cgroups="" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961236 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961242 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961245 2563 flags.go:64] FLAG: --tls-cert-file="" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961247 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961251 2563 flags.go:64] FLAG: --tls-min-version="" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961254 2563 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961257 2563 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961260 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 18:45:39.968900 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961263 2563 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961267 2563 flags.go:64] FLAG: --v="2" Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961271 2563 flags.go:64] FLAG: --version="false" Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961275 2563 flags.go:64] FLAG: --vmodule="" Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961279 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.961282 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961404 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961408 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961413 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961417 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961420 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961422 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961425 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961428 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961431 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961433 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961436 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961439 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961441 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961444 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961447 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:39.969606 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961450 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961453 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961455 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961458 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961461 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961464 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961466 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961469 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961471 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961474 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961477 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961481 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961484 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961487 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961490 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961493 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961496 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961499 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961501 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961505 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:39.970147 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961510 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961513 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961516 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961518 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961521 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961524 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961526 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961529 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961534 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961537 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961540 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961543 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961546 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961549 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961551 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961554 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961557 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961560 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961562 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:39.970653 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961565 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961568 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961570 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961573 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961576 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961578 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961581 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961584 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961587 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961590 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961593 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961596 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961600 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961604 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961608 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961612 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961615 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961618 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961621 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:39.971143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961624 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961627 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961630 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961632 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961636 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961639 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961642 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961645 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961648 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961651 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961654 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961657 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961660 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961663 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961665 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961668 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961671 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961673 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961676 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961678 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:39.971646 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961681 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961684 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961686 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961689 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961692 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.961696 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.962449 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.969546 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.969563 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.970268 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.970276 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970341 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970346 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970350 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970354 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:39.972187 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970357 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970360 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970363 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970366 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970370 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970374 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970377 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970381 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970384 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970387 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970390 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970392 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970395 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970398 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970400 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970403 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970406 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970408 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970411 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970414 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:39.972570 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970416 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970419 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970421 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970425 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970427 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970431 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970434 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970437 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970440 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970443 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970446 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970449 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970452 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970454 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970457 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970459 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970462 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970465 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970468 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970470 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:39.973065 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970473 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970475 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970478 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970481 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970483 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970486 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970488 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970491 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970495 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970499 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970501 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970504 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970507 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970510 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970513 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970517 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970519 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970523 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970526 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970529 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:39.973587 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970532 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970535 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970538 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970541 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970543 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970547 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970549 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970552 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970554 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970557 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970559 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970562 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970564 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970567 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970570 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970572 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970575 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970578 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970580 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970584 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:39.974091 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970588 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970591 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970593 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970596 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970598 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970601 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970604 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970607 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970609 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970612 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970615 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970618 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970621 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970624 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970626 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:39.974682 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.970632 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970763 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970769 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970773 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970776 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970780 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970783 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970786 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970788 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970791 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970794 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970796 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970799 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970801 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970804 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970807 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970810 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970813 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970815 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970818 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 18:45:39.975101 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970820 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970823 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970826 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970844 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970851 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970857 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970863 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970866 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970869 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970873 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970876 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970879 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970882 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970884 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970887 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970889 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970892 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970895 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970897 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 18:45:39.975707 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970900 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970902 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970905 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970908 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970910 2563 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970915 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970918 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970921 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970924 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970927 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970930 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970933 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970936 2563 feature_gate.go:328] unrecognized feature gate: Example Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970938 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970941 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970944 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970946 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970949 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970952 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 18:45:39.976196 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970956 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970958 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970961 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970964 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970967 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970969 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970972 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970975 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970977 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970980 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970982 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970986 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970988 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970991 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970993 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970996 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.970999 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971001 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971003 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971006 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 18:45:39.976975 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971009 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971012 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971014 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971017 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971019 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971022 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971025 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971027 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971030 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971033 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971035 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971038 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971042 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971044 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971047 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971049 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971052 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971055 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971058 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971060 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 18:45:39.977666 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971063 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 18:45:39.978351 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:39.971065 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 18:45:39.978351 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.971071 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 18:45:39.978351 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.971214 2563 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 18:45:39.978351 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.973230 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 18:45:39.978351 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.974401 2563 server.go:1019] "Starting client certificate rotation" Sep 16 18:45:39.978351 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.974517 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:45:39.978351 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.974577 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 18:45:39.978351 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.976826 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:45:39.978609 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.978447 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 18:45:39.988099 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.988075 2563 log.go:25] "Validated CRI v1 runtime API" Sep 16 18:45:39.992432 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.992411 2563 log.go:25] "Validated CRI v1 image API" Sep 16 18:45:39.993327 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.993301 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 18:45:39.995425 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.995398 2563 fs.go:135] Filesystem UUIDs: map[598e3363-d951-412c-bf0d-70f6ec954e3a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a2582254-c764-4491-b136-75d41aa35dac:/dev/nvme0n1p4] Sep 16 18:45:39.995467 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:39.995425 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 18:45:40.001194 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.001079 2563 manager.go:216] Machine: {Timestamp:2026-09-16 18:45:39.999074802 +0000 UTC m=+0.322767643 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102495 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2109f242649de3380cbdbd620f37f4 SystemUUID:ec2109f2-4264-9de3-380c-bdbd620f37f4 BootID:88847a07-6695-4375-a8d0-c79c62d16b33 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fe:bb:ca:c5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fe:bb:ca:c5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ee:12:fb:62:3b:f2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 18:45:40.001194 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.001192 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 18:45:40.001300 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.001282 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 18:45:40.004867 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.004823 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 18:45:40.005032 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.004870 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 18:45:40.005688 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.005679 2563 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 18:45:40.005719 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.005692 2563 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 18:45:40.005719 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.005700 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:45:40.006384 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.006374 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 18:45:40.007799 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.007786 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 18:45:40.011511 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.011498 2563 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:45:40.011633 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.011623 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 18:45:40.011667 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.011659 2563 kubelet.go:486] "Attempting to sync node with API server" Sep 16 18:45:40.011702 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.011671 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 18:45:40.011702 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.011698 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 18:45:40.011758 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.011713 2563 kubelet.go:398] "Adding apiserver pod source" Sep 16 18:45:40.011758 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.011726 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 18:45:40.012209 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.012196 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:45:40.012250 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.012222 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 18:45:40.014497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.014476 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 18:45:40.016706 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.016688 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 18:45:40.016808 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.016727 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 18:45:40.018260 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018246 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 18:45:40.018335 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018267 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 18:45:40.018335 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018275 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 18:45:40.018335 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018283 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 18:45:40.018335 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018296 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 18:45:40.018335 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018304 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 18:45:40.018335 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018326 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 18:45:40.018335 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018334 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 18:45:40.018560 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018345 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 18:45:40.018560 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018353 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 18:45:40.018560 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018366 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 18:45:40.018560 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.018375 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 18:45:40.023674 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.023656 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 18:45:40.023674 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.023676 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 18:45:40.027594 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.027576 2563 server.go:1268] "Started kubelet" Sep 16 18:45:40.029131 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.028687 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 18:45:40.029131 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.028649 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 18:45:40.029131 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.028927 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 18:45:40.029927 ip-10-0-134-125 systemd[1]: Started Kubernetes Kubelet. Sep 16 18:45:40.031327 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.031291 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 18:45:40.033073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.032731 2563 server.go:310] "Adding debug handlers to kubelet server" Sep 16 18:45:40.033637 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.033532 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 18:45:40.037344 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.037324 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 18:45:40.037896 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.037875 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 18:45:40.037994 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.037948 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:40.038502 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.038485 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 18:45:40.038502 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.038495 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 18:45:40.038620 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.038509 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 18:45:40.038665 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.038635 2563 reconstruct.go:97] "Volume reconstruction finished" Sep 16 18:45:40.038665 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.038645 2563 reconciler.go:29] "Reconciler: start to sync state" Sep 16 18:45:40.038740 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.038730 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 18:45:40.038780 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.038744 2563 factory.go:55] Registering systemd factory Sep 16 18:45:40.038780 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.038752 2563 factory.go:223] Registration of the systemd container factory successfully Sep 16 18:45:40.038991 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.038979 2563 factory.go:153] Registering CRI-O factory Sep 16 18:45:40.039038 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.038993 2563 factory.go:223] Registration of the crio container factory successfully Sep 16 18:45:40.039038 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.039008 2563 factory.go:103] Registering Raw factory Sep 16 18:45:40.039038 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.039017 2563 manager.go:1196] Started watching for new ooms in manager Sep 16 18:45:40.039472 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.039460 2563 manager.go:349] Starting recovery of all containers Sep 16 18:45:40.047285 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.047258 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-125.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 18:45:40.047374 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.047288 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 18:45:40.047454 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.047417 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 16 18:45:40.047514 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.047487 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 18:45:40.047620 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.047597 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 16 18:45:40.048416 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.047364 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-125.ec2.internal.18d5e179c2f19b07 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-125.ec2.internal,UID:ip-10-0-134-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-125.ec2.internal,},FirstTimestamp:2026-09-16 18:45:40.027546375 +0000 UTC m=+0.351239215,LastTimestamp:2026-09-16 18:45:40.027546375 +0000 UTC m=+0.351239215,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-125.ec2.internal,}" Sep 16 18:45:40.052317 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.052302 2563 manager.go:354] Recovery completed Sep 16 18:45:40.055880 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.055819 2563 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 16 18:45:40.057820 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.057801 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:45:40.059122 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.059110 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:40.061250 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.061224 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:40.061322 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.061261 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:40.061322 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.061276 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:40.062069 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.062049 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 18:45:40.062069 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.062065 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 18:45:40.062179 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.062088 2563 state_mem.go:36] "Initialized new in-memory state store" Sep 16 18:45:40.062415 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.062392 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:40.064063 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.064042 2563 policy_none.go:49] "None policy: Start" Sep 16 18:45:40.064172 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.064082 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 18:45:40.064172 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.064096 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 18:45:40.064779 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.064757 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:40.064868 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.064803 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:40.064868 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.064820 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:40.065310 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.065295 2563 policy_none.go:47] "Start" Sep 16 18:45:40.075326 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.075248 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-125.ec2.internal.18d5e179c4f3c8ab default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-125.ec2.internal,UID:ip-10-0-134-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-125.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-125.ec2.internal,},FirstTimestamp:2026-09-16 18:45:40.061243563 +0000 UTC m=+0.384936405,LastTimestamp:2026-09-16 18:45:40.061243563 +0000 UTC m=+0.384936405,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-125.ec2.internal,}" Sep 16 18:45:40.085210 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.085103 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-125.ec2.internal.18d5e179c4f425b4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-125.ec2.internal,UID:ip-10-0-134-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-125.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-125.ec2.internal,},FirstTimestamp:2026-09-16 18:45:40.06126738 +0000 UTC m=+0.384960222,LastTimestamp:2026-09-16 18:45:40.06126738 +0000 UTC m=+0.384960222,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-125.ec2.internal,}" Sep 16 18:45:40.088064 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.087993 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-125.ec2.internal.18d5e179c4f45d41 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-125.ec2.internal,UID:ip-10-0-134-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-125.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-125.ec2.internal,},FirstTimestamp:2026-09-16 18:45:40.061281601 +0000 UTC m=+0.384974443,LastTimestamp:2026-09-16 18:45:40.061281601 +0000 UTC m=+0.384974443,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-125.ec2.internal,}" Sep 16 18:45:40.096576 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.096559 2563 manager.go:337] "Starting Device Plugin manager" Sep 16 18:45:40.096668 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.096604 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 18:45:40.096668 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.096617 2563 server.go:87] "Starting device plugin registration server" Sep 16 18:45:40.097059 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.097045 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 18:45:40.097109 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.097068 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 18:45:40.099089 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.097260 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 18:45:40.099089 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.097297 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hnw8n" Sep 16 18:45:40.099089 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.097359 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 18:45:40.099089 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.097365 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 18:45:40.099089 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.097994 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 18:45:40.099089 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.098040 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:40.101820 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.101753 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-125.ec2.internal.18d5e179c4f3c8ab\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-125.ec2.internal.18d5e179c4f3c8ab default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-125.ec2.internal,UID:ip-10-0-134-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-125.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-125.ec2.internal,},FirstTimestamp:2026-09-16 18:45:40.061243563 +0000 UTC m=+0.384936405,LastTimestamp:2026-09-16 18:45:40.064779385 +0000 UTC m=+0.388472226,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-125.ec2.internal,}" Sep 16 18:45:40.107073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.107057 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hnw8n" Sep 16 18:45:40.112521 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.112497 2563 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2592 into the system container \"/system.slice\": " Sep 16 18:45:40.198111 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.198066 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:40.199205 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.199185 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:40.199292 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.199219 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:40.199292 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.199233 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:40.199292 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.199262 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.207955 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.207869 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.207955 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.207897 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-125.ec2.internal\": node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:40.230180 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.230153 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:40.329319 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.329264 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 18:45:40.330523 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.330494 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 18:45:40.330523 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.330511 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:40.330678 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.330547 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 18:45:40.330678 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.330573 2563 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 18:45:40.330678 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.330671 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 18:45:40.334739 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.334720 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:40.431582 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.431542 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:40.431582 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.431560 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-134-125.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal"] Sep 16 18:45:40.431782 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.431654 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:40.432686 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.432670 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:40.432782 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.432706 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:40.432782 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.432720 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:40.433983 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.433966 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:40.434166 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.434152 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.434227 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.434220 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:40.434782 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.434763 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:40.434782 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.434779 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:40.434938 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.434793 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:40.434938 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.434801 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:40.434938 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.434804 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:40.434938 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.434812 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:40.435961 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.435948 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.436019 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.435974 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 18:45:40.436693 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.436676 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientMemory" Sep 16 18:45:40.436788 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.436707 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 18:45:40.436788 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.436723 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeHasSufficientPID" Sep 16 18:45:40.439684 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.439665 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/153bf23bf7549998ce569e477f0251ee-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.ec2.internal\" (UID: \"153bf23bf7549998ce569e477f0251ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.439783 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.439689 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/941dab79fa0f2bb2b6d92051cde9534f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal\" (UID: \"941dab79fa0f2bb2b6d92051cde9534f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.439783 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.439706 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941dab79fa0f2bb2b6d92051cde9534f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal\" (UID: \"941dab79fa0f2bb2b6d92051cde9534f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.453548 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.453520 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-125.ec2.internal\" not found" node="ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.457304 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.457284 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-125.ec2.internal\" not found" node="ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.531752 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.531658 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:40.539956 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.539929 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/153bf23bf7549998ce569e477f0251ee-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.ec2.internal\" (UID: \"153bf23bf7549998ce569e477f0251ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.540049 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.539957 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/941dab79fa0f2bb2b6d92051cde9534f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal\" (UID: \"941dab79fa0f2bb2b6d92051cde9534f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.540049 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.539974 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941dab79fa0f2bb2b6d92051cde9534f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal\" (UID: \"941dab79fa0f2bb2b6d92051cde9534f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.540049 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.540034 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/941dab79fa0f2bb2b6d92051cde9534f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal\" (UID: \"941dab79fa0f2bb2b6d92051cde9534f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.540049 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.540040 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/153bf23bf7549998ce569e477f0251ee-config\") pod \"kube-apiserver-proxy-ip-10-0-134-125.ec2.internal\" (UID: \"153bf23bf7549998ce569e477f0251ee\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.540173 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.540056 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/941dab79fa0f2bb2b6d92051cde9534f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal\" (UID: \"941dab79fa0f2bb2b6d92051cde9534f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.632169 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.632126 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:40.732260 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.732226 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:40.755740 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.755713 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.760631 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.760611 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" Sep 16 18:45:40.833242 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.833153 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:40.933775 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:40.933735 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:40.936511 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.936491 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:40.973934 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.973899 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 18:45:40.974433 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.974050 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 18:45:40.974433 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:40.974086 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 18:45:41.034485 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:41.034455 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-125.ec2.internal\" not found" Sep 16 18:45:41.079365 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:41.079333 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:41.110240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:41.110145 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:40:40 +0000 UTC" deadline="2028-03-21 04:22:49.880036763 +0000 UTC" Sep 16 18:45:41.110240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:41.110197 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13233h37m8.769844366s" Sep 16 18:45:41.138408 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:41.138381 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.ec2.internal" Sep 16 18:45:41.152495 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:41.152466 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:45:41.154269 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:41.154252 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" Sep 16 18:45:41.165738 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:41.165709 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 18:45:41.386979 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:41.386941 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod941dab79fa0f2bb2b6d92051cde9534f.slice/crio-ed25801f78c3e714e859596bcb794ae4b3d76f22cfc0c33593b98ebed2e48e12 WatchSource:0}: Error finding container ed25801f78c3e714e859596bcb794ae4b3d76f22cfc0c33593b98ebed2e48e12: Status 404 returned error can't find the container with id ed25801f78c3e714e859596bcb794ae4b3d76f22cfc0c33593b98ebed2e48e12 Sep 16 18:45:41.387330 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:41.387307 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153bf23bf7549998ce569e477f0251ee.slice/crio-40432aebe8c0b077c6fcc9a8436ece84885cbcace46c6b863b7197c1fa6c8d50 WatchSource:0}: Error finding container 40432aebe8c0b077c6fcc9a8436ece84885cbcace46c6b863b7197c1fa6c8d50: Status 404 returned error can't find the container with id 40432aebe8c0b077c6fcc9a8436ece84885cbcace46c6b863b7197c1fa6c8d50 Sep 16 18:45:41.392065 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:41.392048 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:45:41.455234 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:41.455206 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:42.012772 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.012729 2563 apiserver.go:52] "Watching apiserver" Sep 16 18:45:42.020093 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.019023 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 18:45:42.020093 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.019626 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-5qjc7","kube-system/kube-apiserver-proxy-ip-10-0-134-125.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk","openshift-dns/node-resolver-lh5wh","openshift-image-registry/node-ca-h946g","openshift-multus/multus-jprzs","openshift-network-diagnostics/network-check-target-9k5bx","openshift-ovn-kubernetes/ovnkube-node-bx8nl","kube-system/global-pull-secret-syncer-klhb5","kube-system/konnectivity-agent-m5czp","openshift-cluster-node-tuning-operator/tuned-rjhmn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal","openshift-multus/multus-additional-cni-plugins-vjvnd","openshift-multus/network-metrics-daemon-lq68v"] Sep 16 18:45:42.022069 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.021633 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:42.022069 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.021716 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:45:42.024063 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.023267 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.026497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.026201 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-95nbs\"" Sep 16 18:45:42.026497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.026461 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 18:45:42.026671 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.026636 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 18:45:42.026871 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.026825 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 18:45:42.026871 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.026868 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 18:45:42.027029 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.026893 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 18:45:42.027029 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.026942 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.028256 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.028129 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lh5wh" Sep 16 18:45:42.029958 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.029940 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h946g" Sep 16 18:45:42.031534 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.031515 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.033549 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.033219 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:42.033549 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.033348 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:45:42.034534 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.034430 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 18:45:42.035910 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.035219 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.036163 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.036613 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qc8ch\"" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.036705 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-75hml\"" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.036876 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.037118 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.037140 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.037155 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.037338 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5qjc7" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.037360 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.037471 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.037763 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.038117 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8dqkm\"" Sep 16 18:45:42.038450 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.038311 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hdqkd\"" Sep 16 18:45:42.039104 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.038573 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:42.039989 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.039577 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:45:42.040556 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.040086 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:45:42.042088 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.042061 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.042961 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.042882 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:45:42.043065 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.042961 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 18:45:42.043065 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.043005 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pg8wv\"" Sep 16 18:45:42.043306 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.043289 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 18:45:42.043597 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.043579 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 18:45:42.043670 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.043596 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.043670 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.043579 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jzwtt\"" Sep 16 18:45:42.043924 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.043885 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 18:45:42.047023 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047000 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-var-lib-cni-multus\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.047116 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-hostroot\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.047184 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cr5q\" (UniqueName: \"kubernetes.io/projected/01d77fab-c94e-46ea-b0e8-1b2b02f5d977-kube-api-access-2cr5q\") pod \"node-ca-h946g\" (UID: \"01d77fab-c94e-46ea-b0e8-1b2b02f5d977\") " pod="openshift-image-registry/node-ca-h946g" Sep 16 18:45:42.047184 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047173 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-system-cni-dir\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.047284 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047197 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-multus-cni-dir\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.047284 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047211 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 18:45:42.047284 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047279 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 18:45:42.047485 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047408 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 18:45:42.047659 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047640 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 18:45:42.047763 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047749 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-6tjbx\"" Sep 16 18:45:42.048089 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047226 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-var-lib-kubelet\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.048089 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047908 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 18:45:42.048089 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.047942 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:45:42.048089 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.048045 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 18:45:42.048318 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.048192 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 18:45:42.048643 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.048536 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kpj5f\"" Sep 16 18:45:42.048915 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.048871 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f741ffaa-797b-4f41-ac31-6da5fe63d9bd-agent-certs\") pod \"konnectivity-agent-m5czp\" (UID: \"f741ffaa-797b-4f41-ac31-6da5fe63d9bd\") " pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:45:42.049001 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.048920 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-device-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.049001 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.048950 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2231582-bd0c-440b-9841-bd4740794c8c-tmp-dir\") pod \"node-resolver-lh5wh\" (UID: \"f2231582-bd0c-440b-9841-bd4740794c8c\") " pod="openshift-dns/node-resolver-lh5wh" Sep 16 18:45:42.049001 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.048974 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01d77fab-c94e-46ea-b0e8-1b2b02f5d977-host\") pod \"node-ca-h946g\" (UID: \"01d77fab-c94e-46ea-b0e8-1b2b02f5d977\") " pod="openshift-image-registry/node-ca-h946g" Sep 16 18:45:42.049001 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.048996 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-etc-kubernetes\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.049204 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049021 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnkjr\" (UniqueName: \"kubernetes.io/projected/650deff9-b845-4ef3-acad-6bdee9667025-kube-api-access-pnkjr\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:42.049204 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049045 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-socket-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.049204 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049070 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc8dn\" (UniqueName: \"kubernetes.io/projected/f038ea97-8f13-448a-a62e-547ae8ba0c67-kube-api-access-tc8dn\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.049204 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049093 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-dbus\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:42.049204 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049117 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a70859e-eddb-45d0-b551-7c79611c1d90-host-slash\") pod \"iptables-alerter-5qjc7\" (UID: \"8a70859e-eddb-45d0-b551-7c79611c1d90\") " pod="openshift-network-operator/iptables-alerter-5qjc7" Sep 16 18:45:42.049204 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049140 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5qkn\" (UniqueName: \"kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn\") pod \"network-check-target-9k5bx\" (UID: \"94040f40-6ebc-4c73-9280-83b5e973a840\") " pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:42.049204 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049199 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q6pf\" (UniqueName: \"kubernetes.io/projected/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-kube-api-access-9q6pf\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.049525 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049238 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-cnibin\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.049525 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049273 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-run-k8s-cni-cncf-io\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.049525 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049345 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-sys-fs\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.049525 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049413 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2231582-bd0c-440b-9841-bd4740794c8c-hosts-file\") pod \"node-resolver-lh5wh\" (UID: \"f2231582-bd0c-440b-9841-bd4740794c8c\") " pod="openshift-dns/node-resolver-lh5wh" Sep 16 18:45:42.049525 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049443 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01d77fab-c94e-46ea-b0e8-1b2b02f5d977-serviceca\") pod \"node-ca-h946g\" (UID: \"01d77fab-c94e-46ea-b0e8-1b2b02f5d977\") " pod="openshift-image-registry/node-ca-h946g" Sep 16 18:45:42.049525 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049469 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:42.049525 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049509 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049531 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97e78433-797a-47b6-9b23-798ddb52e025-cni-binary-copy\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049554 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-run-netns\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049577 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-var-lib-cni-bin\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049611 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-registration-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049640 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-etc-selinux\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049667 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tw2f\" (UniqueName: \"kubernetes.io/projected/f2231582-bd0c-440b-9841-bd4740794c8c-kube-api-access-8tw2f\") pod \"node-resolver-lh5wh\" (UID: \"f2231582-bd0c-440b-9841-bd4740794c8c\") " pod="openshift-dns/node-resolver-lh5wh" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049694 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76pz4\" (UniqueName: \"kubernetes.io/projected/8a70859e-eddb-45d0-b551-7c79611c1d90-kube-api-access-76pz4\") pod \"iptables-alerter-5qjc7\" (UID: \"8a70859e-eddb-45d0-b551-7c79611c1d90\") " pod="openshift-network-operator/iptables-alerter-5qjc7" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049716 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-cni-binary-copy\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049752 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049777 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f741ffaa-797b-4f41-ac31-6da5fe63d9bd-konnectivity-ca\") pod \"konnectivity-agent-m5czp\" (UID: \"f741ffaa-797b-4f41-ac31-6da5fe63d9bd\") " pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049801 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-cnibin\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049825 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-multus-socket-dir-parent\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.049914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049899 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97e78433-797a-47b6-9b23-798ddb52e025-multus-daemon-config\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.050457 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049922 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-run-multus-certs\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.050457 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049949 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-kubelet-config\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:42.050457 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.049971 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.050457 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.050012 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-multus-conf-dir\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.050457 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.050045 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscxk\" (UniqueName: \"kubernetes.io/projected/97e78433-797a-47b6-9b23-798ddb52e025-kube-api-access-sscxk\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.050457 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.050074 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:42.050457 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.050101 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.050457 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.050124 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a70859e-eddb-45d0-b551-7c79611c1d90-iptables-alerter-script\") pod \"iptables-alerter-5qjc7\" (UID: \"8a70859e-eddb-45d0-b551-7c79611c1d90\") " pod="openshift-network-operator/iptables-alerter-5qjc7" Sep 16 18:45:42.050457 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.050147 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-system-cni-dir\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.053843 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.053572 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 18:45:42.074518 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.074495 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ltpph" Sep 16 18:45:42.087010 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.086956 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ltpph" Sep 16 18:45:42.118365 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.118210 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:42.140062 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.140029 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150343 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01d77fab-c94e-46ea-b0e8-1b2b02f5d977-host\") pod \"node-ca-h946g\" (UID: \"01d77fab-c94e-46ea-b0e8-1b2b02f5d977\") " pod="openshift-image-registry/node-ca-h946g" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150387 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-run-netns\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97e78433-797a-47b6-9b23-798ddb52e025-multus-daemon-config\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150458 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-sysctl-conf\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150475 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/01d77fab-c94e-46ea-b0e8-1b2b02f5d977-host\") pod \"node-ca-h946g\" (UID: \"01d77fab-c94e-46ea-b0e8-1b2b02f5d977\") " pod="openshift-image-registry/node-ca-h946g" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150483 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-systemd\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150548 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-socket-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150572 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-dbus\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150612 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a70859e-eddb-45d0-b551-7c79611c1d90-host-slash\") pod \"iptables-alerter-5qjc7\" (UID: \"8a70859e-eddb-45d0-b551-7c79611c1d90\") " pod="openshift-network-operator/iptables-alerter-5qjc7" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150677 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a70859e-eddb-45d0-b551-7c79611c1d90-host-slash\") pod \"iptables-alerter-5qjc7\" (UID: \"8a70859e-eddb-45d0-b551-7c79611c1d90\") " pod="openshift-network-operator/iptables-alerter-5qjc7" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qkn\" (UniqueName: \"kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn\") pod \"network-check-target-9k5bx\" (UID: \"94040f40-6ebc-4c73-9280-83b5e973a840\") " pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150745 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-dbus\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150739 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-run-netns\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150779 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2231582-bd0c-440b-9841-bd4740794c8c-hosts-file\") pod \"node-resolver-lh5wh\" (UID: \"f2231582-bd0c-440b-9841-bd4740794c8c\") " pod="openshift-dns/node-resolver-lh5wh" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150785 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-run-netns\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150787 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-socket-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150802 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01d77fab-c94e-46ea-b0e8-1b2b02f5d977-serviceca\") pod \"node-ca-h946g\" (UID: \"01d77fab-c94e-46ea-b0e8-1b2b02f5d977\") " pod="openshift-image-registry/node-ca-h946g" Sep 16 18:45:42.150883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150853 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f2231582-bd0c-440b-9841-bd4740794c8c-hosts-file\") pod \"node-resolver-lh5wh\" (UID: \"f2231582-bd0c-440b-9841-bd4740794c8c\") " pod="openshift-dns/node-resolver-lh5wh" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150954 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97e78433-797a-47b6-9b23-798ddb52e025-cni-binary-copy\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.150979 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-var-lib-cni-bin\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151010 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz7gn\" (UniqueName: \"kubernetes.io/projected/b2648669-4bd6-4a83-92ef-b694ca40e42c-kube-api-access-qz7gn\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151035 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ae005ac-ab99-433a-bd5c-7b8193356ffb-ovn-node-metrics-cert\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151057 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxmf\" (UniqueName: \"kubernetes.io/projected/4ae005ac-ab99-433a-bd5c-7b8193356ffb-kube-api-access-frxmf\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151090 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-etc-selinux\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151116 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76pz4\" (UniqueName: \"kubernetes.io/projected/8a70859e-eddb-45d0-b551-7c79611c1d90-kube-api-access-76pz4\") pod \"iptables-alerter-5qjc7\" (UID: \"8a70859e-eddb-45d0-b551-7c79611c1d90\") " pod="openshift-network-operator/iptables-alerter-5qjc7" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151111 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-var-lib-cni-bin\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151150 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/97e78433-797a-47b6-9b23-798ddb52e025-multus-daemon-config\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151176 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-etc-selinux\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151205 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-cni-binary-copy\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151232 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f741ffaa-797b-4f41-ac31-6da5fe63d9bd-konnectivity-ca\") pod \"konnectivity-agent-m5czp\" (UID: \"f741ffaa-797b-4f41-ac31-6da5fe63d9bd\") " pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151283 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-tuned\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151308 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-run-ovn-kubernetes\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151334 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-cnibin\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151356 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-multus-socket-dir-parent\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.151967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151381 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-sysctl-d\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151422 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-lib-modules\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151450 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-multus-socket-dir-parent\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151454 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ae005ac-ab99-433a-bd5c-7b8193356ffb-ovnkube-config\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151492 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/97e78433-797a-47b6-9b23-798ddb52e025-cni-binary-copy\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151502 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-cnibin\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151513 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ae005ac-ab99-433a-bd5c-7b8193356ffb-ovnkube-script-lib\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151565 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-kubelet-config\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151592 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151617 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-multus-conf-dir\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151624 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-kubelet-config\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151642 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscxk\" (UniqueName: \"kubernetes.io/projected/97e78433-797a-47b6-9b23-798ddb52e025-kube-api-access-sscxk\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151667 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-systemd-units\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151693 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-var-lib-openvswitch\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151716 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-cni-netd\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151773 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/01d77fab-c94e-46ea-b0e8-1b2b02f5d977-serviceca\") pod \"node-ca-h946g\" (UID: \"01d77fab-c94e-46ea-b0e8-1b2b02f5d977\") " pod="openshift-image-registry/node-ca-h946g" Sep 16 18:45:42.152723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151809 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-system-cni-dir\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151809 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151809 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-cni-binary-copy\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151771 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-system-cni-dir\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151855 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-multus-conf-dir\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151873 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-sys\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151900 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-var-lib-kubelet\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151918 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151925 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-slash\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151921 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f741ffaa-797b-4f41-ac31-6da5fe63d9bd-konnectivity-ca\") pod \"konnectivity-agent-m5czp\" (UID: \"f741ffaa-797b-4f41-ac31-6da5fe63d9bd\") " pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.151977 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152000 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-multus-cni-dir\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f741ffaa-797b-4f41-ac31-6da5fe63d9bd-agent-certs\") pod \"konnectivity-agent-m5czp\" (UID: \"f741ffaa-797b-4f41-ac31-6da5fe63d9bd\") " pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152094 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-multus-cni-dir\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2648669-4bd6-4a83-92ef-b694ca40e42c-tmp\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152156 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-kubelet\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152183 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-device-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.153483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152223 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2231582-bd0c-440b-9841-bd4740794c8c-tmp-dir\") pod \"node-resolver-lh5wh\" (UID: \"f2231582-bd0c-440b-9841-bd4740794c8c\") " pod="openshift-dns/node-resolver-lh5wh" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-etc-kubernetes\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152270 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnkjr\" (UniqueName: \"kubernetes.io/projected/650deff9-b845-4ef3-acad-6bdee9667025-kube-api-access-pnkjr\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152294 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-run-multus-certs\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152314 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc8dn\" (UniqueName: \"kubernetes.io/projected/f038ea97-8f13-448a-a62e-547ae8ba0c67-kube-api-access-tc8dn\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152336 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q6pf\" (UniqueName: \"kubernetes.io/projected/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-kube-api-access-9q6pf\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152362 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-cnibin\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152385 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-run-k8s-cni-cncf-io\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152395 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-run-multus-certs\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152410 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-run\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152433 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-host\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152432 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-etc-kubernetes\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152563 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-cni-bin\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152559 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2231582-bd0c-440b-9841-bd4740794c8c-tmp-dir\") pod \"node-resolver-lh5wh\" (UID: \"f2231582-bd0c-440b-9841-bd4740794c8c\") " pod="openshift-dns/node-resolver-lh5wh" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152591 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-sys-fs\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152602 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-run-k8s-cni-cncf-io\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152615 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:42.154156 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152361 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-device-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152642 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152669 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-sysconfig\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152684 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-cnibin\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152693 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-registration-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152735 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-sys-fs\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152738 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.152754 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152758 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tw2f\" (UniqueName: \"kubernetes.io/projected/f2231582-bd0c-440b-9841-bd4740794c8c-kube-api-access-8tw2f\") pod \"node-resolver-lh5wh\" (UID: \"f2231582-bd0c-440b-9841-bd4740794c8c\") " pod="openshift-dns/node-resolver-lh5wh" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152785 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152801 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f038ea97-8f13-448a-a62e-547ae8ba0c67-registration-dir\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152811 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-kubernetes\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.152880 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret podName:0fed9f62-c1e7-419f-b5d2-85d3214e9d8e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:42.652858753 +0000 UTC m=+2.976551600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret") pod "global-pull-secret-syncer-klhb5" (UID: "0fed9f62-c1e7-419f-b5d2-85d3214e9d8e") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152904 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-run-openvswitch\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152935 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152960 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-modprobe-d\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.152990 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-run-ovn\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.154932 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.153026 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153028 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-log-socket\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153057 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a70859e-eddb-45d0-b551-7c79611c1d90-iptables-alerter-script\") pod \"iptables-alerter-5qjc7\" (UID: \"8a70859e-eddb-45d0-b551-7c79611c1d90\") " pod="openshift-network-operator/iptables-alerter-5qjc7" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.153087 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs podName:650deff9-b845-4ef3-acad-6bdee9667025 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:42.65307023 +0000 UTC m=+2.976763067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs") pod "network-metrics-daemon-lq68v" (UID: "650deff9-b845-4ef3-acad-6bdee9667025") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153107 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-var-lib-cni-multus\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153159 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-var-lib-cni-multus\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153177 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-hostroot\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153201 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-run-systemd\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153219 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ae005ac-ab99-433a-bd5c-7b8193356ffb-env-overrides\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153241 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cr5q\" (UniqueName: \"kubernetes.io/projected/01d77fab-c94e-46ea-b0e8-1b2b02f5d977-kube-api-access-2cr5q\") pod \"node-ca-h946g\" (UID: \"01d77fab-c94e-46ea-b0e8-1b2b02f5d977\") " pod="openshift-image-registry/node-ca-h946g" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153263 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-system-cni-dir\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153283 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-var-lib-kubelet\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153303 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-etc-openvswitch\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-node-log\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153374 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153445 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-host-var-lib-kubelet\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153461 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-system-cni-dir\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.155790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153494 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/97e78433-797a-47b6-9b23-798ddb52e025-hostroot\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.156414 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153588 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.156414 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.153588 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a70859e-eddb-45d0-b551-7c79611c1d90-iptables-alerter-script\") pod \"iptables-alerter-5qjc7\" (UID: \"8a70859e-eddb-45d0-b551-7c79611c1d90\") " pod="openshift-network-operator/iptables-alerter-5qjc7" Sep 16 18:45:42.157397 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.157375 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f741ffaa-797b-4f41-ac31-6da5fe63d9bd-agent-certs\") pod \"konnectivity-agent-m5czp\" (UID: \"f741ffaa-797b-4f41-ac31-6da5fe63d9bd\") " pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:45:42.164922 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.164879 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:42.164922 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.164909 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:42.164922 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.164920 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d5qkn for pod openshift-network-diagnostics/network-check-target-9k5bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:42.165147 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.164988 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn podName:94040f40-6ebc-4c73-9280-83b5e973a840 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:42.664972204 +0000 UTC m=+2.988665044 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qkn" (UniqueName: "kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn") pod "network-check-target-9k5bx" (UID: "94040f40-6ebc-4c73-9280-83b5e973a840") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:42.168187 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.168157 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscxk\" (UniqueName: \"kubernetes.io/projected/97e78433-797a-47b6-9b23-798ddb52e025-kube-api-access-sscxk\") pod \"multus-jprzs\" (UID: \"97e78433-797a-47b6-9b23-798ddb52e025\") " pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.168735 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.168707 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76pz4\" (UniqueName: \"kubernetes.io/projected/8a70859e-eddb-45d0-b551-7c79611c1d90-kube-api-access-76pz4\") pod \"iptables-alerter-5qjc7\" (UID: \"8a70859e-eddb-45d0-b551-7c79611c1d90\") " pod="openshift-network-operator/iptables-alerter-5qjc7" Sep 16 18:45:42.174805 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.174775 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tw2f\" (UniqueName: \"kubernetes.io/projected/f2231582-bd0c-440b-9841-bd4740794c8c-kube-api-access-8tw2f\") pod \"node-resolver-lh5wh\" (UID: \"f2231582-bd0c-440b-9841-bd4740794c8c\") " pod="openshift-dns/node-resolver-lh5wh" Sep 16 18:45:42.175546 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.175521 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q6pf\" (UniqueName: \"kubernetes.io/projected/003ff6ca-4a01-4b09-98ea-17bf09a51ffe-kube-api-access-9q6pf\") pod \"multus-additional-cni-plugins-vjvnd\" (UID: \"003ff6ca-4a01-4b09-98ea-17bf09a51ffe\") " pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.175937 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.175890 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc8dn\" (UniqueName: \"kubernetes.io/projected/f038ea97-8f13-448a-a62e-547ae8ba0c67-kube-api-access-tc8dn\") pod \"aws-ebs-csi-driver-node-sgwmk\" (UID: \"f038ea97-8f13-448a-a62e-547ae8ba0c67\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.176078 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.176059 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnkjr\" (UniqueName: \"kubernetes.io/projected/650deff9-b845-4ef3-acad-6bdee9667025-kube-api-access-pnkjr\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:42.177930 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.177911 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cr5q\" (UniqueName: \"kubernetes.io/projected/01d77fab-c94e-46ea-b0e8-1b2b02f5d977-kube-api-access-2cr5q\") pod \"node-ca-h946g\" (UID: \"01d77fab-c94e-46ea-b0e8-1b2b02f5d977\") " pod="openshift-image-registry/node-ca-h946g" Sep 16 18:45:42.253745 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253713 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ae005ac-ab99-433a-bd5c-7b8193356ffb-ovn-node-metrics-cert\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.253745 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxmf\" (UniqueName: \"kubernetes.io/projected/4ae005ac-ab99-433a-bd5c-7b8193356ffb-kube-api-access-frxmf\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.253994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253765 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-tuned\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.253994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253780 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-run-ovn-kubernetes\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.253994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253803 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-sysctl-d\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.253994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253825 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-lib-modules\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.253994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253873 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ae005ac-ab99-433a-bd5c-7b8193356ffb-ovnkube-config\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.253994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253902 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ae005ac-ab99-433a-bd5c-7b8193356ffb-ovnkube-script-lib\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.253994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253919 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-systemd-units\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.253994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253934 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-var-lib-openvswitch\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.253994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253955 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-cni-netd\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.253994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253975 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-sys\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.253994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.253993 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-var-lib-kubelet\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254016 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-slash\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254040 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254068 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2648669-4bd6-4a83-92ef-b694ca40e42c-tmp\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254086 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-kubelet\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-run\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254136 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-host\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254144 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-var-lib-openvswitch\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254175 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-cni-bin\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254208 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-systemd-units\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254211 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-sysconfig\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254240 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-run-ovn-kubernetes\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254252 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-kubernetes\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254268 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-sysconfig\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254277 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-run-openvswitch\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254281 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-lib-modules\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254310 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-run-openvswitch\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254325 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-sysctl-d\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.254497 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254329 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-modprobe-d\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254368 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-run-ovn\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254384 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-kubernetes\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254395 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-log-socket\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254332 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-sys\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254421 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-cni-netd\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254423 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-run-systemd\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254448 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-run-ovn\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254456 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-run-systemd\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254459 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ae005ac-ab99-433a-bd5c-7b8193356ffb-env-overrides\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254476 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-log-socket\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254494 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-var-lib-kubelet\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254508 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254544 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-modprobe-d\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254567 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-run\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254600 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-etc-openvswitch\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254628 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-node-log\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254628 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-cni-bin\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.255223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254676 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-kubelet\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254689 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-slash\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254701 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4ae005ac-ab99-433a-bd5c-7b8193356ffb-ovnkube-script-lib\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254709 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-etc-openvswitch\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254727 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-node-log\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254740 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4ae005ac-ab99-433a-bd5c-7b8193356ffb-ovnkube-config\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254752 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-host\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254763 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-run-netns\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254796 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-sysctl-conf\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254822 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-systemd\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254799 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4ae005ac-ab99-433a-bd5c-7b8193356ffb-host-run-netns\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254889 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-systemd\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254897 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz7gn\" (UniqueName: \"kubernetes.io/projected/b2648669-4bd6-4a83-92ef-b694ca40e42c-kube-api-access-qz7gn\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254932 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-sysctl-conf\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.256073 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.254999 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4ae005ac-ab99-433a-bd5c-7b8193356ffb-env-overrides\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.256893 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.256873 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b2648669-4bd6-4a83-92ef-b694ca40e42c-etc-tuned\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.257183 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.257166 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ae005ac-ab99-433a-bd5c-7b8193356ffb-ovn-node-metrics-cert\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.257654 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.257634 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2648669-4bd6-4a83-92ef-b694ca40e42c-tmp\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.266338 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.266267 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz7gn\" (UniqueName: \"kubernetes.io/projected/b2648669-4bd6-4a83-92ef-b694ca40e42c-kube-api-access-qz7gn\") pod \"tuned-rjhmn\" (UID: \"b2648669-4bd6-4a83-92ef-b694ca40e42c\") " pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.266465 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.266404 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxmf\" (UniqueName: \"kubernetes.io/projected/4ae005ac-ab99-433a-bd5c-7b8193356ffb-kube-api-access-frxmf\") pod \"ovnkube-node-bx8nl\" (UID: \"4ae005ac-ab99-433a-bd5c-7b8193356ffb\") " pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.337777 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.337713 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" event={"ID":"941dab79fa0f2bb2b6d92051cde9534f","Type":"ContainerStarted","Data":"ed25801f78c3e714e859596bcb794ae4b3d76f22cfc0c33593b98ebed2e48e12"} Sep 16 18:45:42.339095 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.338808 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vjvnd" Sep 16 18:45:42.340630 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.340588 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.ec2.internal" event={"ID":"153bf23bf7549998ce569e477f0251ee","Type":"ContainerStarted","Data":"40432aebe8c0b077c6fcc9a8436ece84885cbcace46c6b863b7197c1fa6c8d50"} Sep 16 18:45:42.347962 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.347926 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" Sep 16 18:45:42.360726 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.360699 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lh5wh" Sep 16 18:45:42.366399 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.366372 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h946g" Sep 16 18:45:42.374134 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.374111 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jprzs" Sep 16 18:45:42.380943 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.380918 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5qjc7" Sep 16 18:45:42.392588 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.392560 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:45:42.399282 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.399259 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" Sep 16 18:45:42.406193 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.406169 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:45:42.657996 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.657915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:42.657996 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.657970 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:42.658230 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.658086 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:42.658230 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.658177 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret podName:0fed9f62-c1e7-419f-b5d2-85d3214e9d8e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:43.65815466 +0000 UTC m=+3.981847503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret") pod "global-pull-secret-syncer-klhb5" (UID: "0fed9f62-c1e7-419f-b5d2-85d3214e9d8e") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:42.658230 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.658086 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:42.658397 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.658249 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs podName:650deff9-b845-4ef3-acad-6bdee9667025 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:43.65823325 +0000 UTC m=+3.981926095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs") pod "network-metrics-daemon-lq68v" (UID: "650deff9-b845-4ef3-acad-6bdee9667025") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:42.758601 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.758562 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qkn\" (UniqueName: \"kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn\") pod \"network-check-target-9k5bx\" (UID: \"94040f40-6ebc-4c73-9280-83b5e973a840\") " pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:42.758765 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.758711 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:42.758765 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.758726 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:42.758765 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.758735 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d5qkn for pod openshift-network-diagnostics/network-check-target-9k5bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:42.758925 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:42.758788 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn podName:94040f40-6ebc-4c73-9280-83b5e973a840 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:43.75877485 +0000 UTC m=+4.082467677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qkn" (UniqueName: "kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn") pod "network-check-target-9k5bx" (UID: "94040f40-6ebc-4c73-9280-83b5e973a840") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:42.787761 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:42.787725 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 18:45:43.089162 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.089051 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:42 +0000 UTC" deadline="2028-04-22 10:56:33.971822622 +0000 UTC" Sep 16 18:45:43.089162 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.089102 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14008h10m50.882723451s" Sep 16 18:45:43.229715 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:43.229373 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2648669_4bd6_4a83_92ef_b694ca40e42c.slice/crio-fc038f5d3175bc3a6e0b2c4826aff5018778ae5ef148e2dbf9385736071c4f03 WatchSource:0}: Error finding container fc038f5d3175bc3a6e0b2c4826aff5018778ae5ef148e2dbf9385736071c4f03: Status 404 returned error can't find the container with id fc038f5d3175bc3a6e0b2c4826aff5018778ae5ef148e2dbf9385736071c4f03 Sep 16 18:45:43.233063 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:43.233037 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e78433_797a_47b6_9b23_798ddb52e025.slice/crio-6653884765df274da92cdb8df8eacc89bbfd01c2475f7f99f5ba36b86a4116cf WatchSource:0}: Error finding container 6653884765df274da92cdb8df8eacc89bbfd01c2475f7f99f5ba36b86a4116cf: Status 404 returned error can't find the container with id 6653884765df274da92cdb8df8eacc89bbfd01c2475f7f99f5ba36b86a4116cf Sep 16 18:45:43.239164 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:43.239118 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d77fab_c94e_46ea_b0e8_1b2b02f5d977.slice/crio-650c3a3756f390373197ff08a2e6ecf5b641be15855bd3ed121a68e43e208201 WatchSource:0}: Error finding container 650c3a3756f390373197ff08a2e6ecf5b641be15855bd3ed121a68e43e208201: Status 404 returned error can't find the container with id 650c3a3756f390373197ff08a2e6ecf5b641be15855bd3ed121a68e43e208201 Sep 16 18:45:43.241208 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:43.240713 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a70859e_eddb_45d0_b551_7c79611c1d90.slice/crio-21decc2d73bca3c24fa410c2875b8fd9368c9698eb55d45cbe86d1f5d0e581cb WatchSource:0}: Error finding container 21decc2d73bca3c24fa410c2875b8fd9368c9698eb55d45cbe86d1f5d0e581cb: Status 404 returned error can't find the container with id 21decc2d73bca3c24fa410c2875b8fd9368c9698eb55d45cbe86d1f5d0e581cb Sep 16 18:45:43.243027 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:43.241517 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf038ea97_8f13_448a_a62e_547ae8ba0c67.slice/crio-eefcfb3d9242b9e9b5200560afe892dfc16ea3ad7aaa8500bfc39842956b5a08 WatchSource:0}: Error finding container eefcfb3d9242b9e9b5200560afe892dfc16ea3ad7aaa8500bfc39842956b5a08: Status 404 returned error can't find the container with id eefcfb3d9242b9e9b5200560afe892dfc16ea3ad7aaa8500bfc39842956b5a08 Sep 16 18:45:43.243027 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:45:43.242751 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003ff6ca_4a01_4b09_98ea_17bf09a51ffe.slice/crio-c14e77e5f224347bc558c37fd399ff05eb59166dd4c3deb19c7a43d82477d52e WatchSource:0}: Error finding container c14e77e5f224347bc558c37fd399ff05eb59166dd4c3deb19c7a43d82477d52e: Status 404 returned error can't find the container with id c14e77e5f224347bc558c37fd399ff05eb59166dd4c3deb19c7a43d82477d52e Sep 16 18:45:43.330999 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.330960 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:43.331179 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:43.331097 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:45:43.343537 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.343458 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lh5wh" event={"ID":"f2231582-bd0c-440b-9841-bd4740794c8c","Type":"ContainerStarted","Data":"349824161dc3fc7acbde81427094ca7a123fa4b7ac7730fd0c7273f5f71bb560"} Sep 16 18:45:43.344804 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.344776 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" event={"ID":"b2648669-4bd6-4a83-92ef-b694ca40e42c","Type":"ContainerStarted","Data":"fc038f5d3175bc3a6e0b2c4826aff5018778ae5ef148e2dbf9385736071c4f03"} Sep 16 18:45:43.346967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.346928 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.ec2.internal" event={"ID":"153bf23bf7549998ce569e477f0251ee","Type":"ContainerStarted","Data":"53de08a7de5ac5c305fe851f380fcdf89129377e0c99b82a0c08c113655693d3"} Sep 16 18:45:43.348237 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.348179 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjvnd" event={"ID":"003ff6ca-4a01-4b09-98ea-17bf09a51ffe","Type":"ContainerStarted","Data":"c14e77e5f224347bc558c37fd399ff05eb59166dd4c3deb19c7a43d82477d52e"} Sep 16 18:45:43.349501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.349477 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" event={"ID":"f038ea97-8f13-448a-a62e-547ae8ba0c67","Type":"ContainerStarted","Data":"eefcfb3d9242b9e9b5200560afe892dfc16ea3ad7aaa8500bfc39842956b5a08"} Sep 16 18:45:43.350791 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.350767 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5qjc7" event={"ID":"8a70859e-eddb-45d0-b551-7c79611c1d90","Type":"ContainerStarted","Data":"21decc2d73bca3c24fa410c2875b8fd9368c9698eb55d45cbe86d1f5d0e581cb"} Sep 16 18:45:43.352155 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.352132 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" event={"ID":"4ae005ac-ab99-433a-bd5c-7b8193356ffb","Type":"ContainerStarted","Data":"0af1c549d524f0eaa6d192497598fd2c77898b5080c1c2f79c4353d45bf37476"} Sep 16 18:45:43.353392 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.353371 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jprzs" event={"ID":"97e78433-797a-47b6-9b23-798ddb52e025","Type":"ContainerStarted","Data":"6653884765df274da92cdb8df8eacc89bbfd01c2475f7f99f5ba36b86a4116cf"} Sep 16 18:45:43.354617 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.354558 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h946g" event={"ID":"01d77fab-c94e-46ea-b0e8-1b2b02f5d977","Type":"ContainerStarted","Data":"650c3a3756f390373197ff08a2e6ecf5b641be15855bd3ed121a68e43e208201"} Sep 16 18:45:43.355833 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.355809 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5czp" event={"ID":"f741ffaa-797b-4f41-ac31-6da5fe63d9bd","Type":"ContainerStarted","Data":"ccd16e42790686e565917578336f30efd5a7598641ade18aa3172802a9c79e17"} Sep 16 18:45:43.363030 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.362982 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-125.ec2.internal" podStartSLOduration=2.362970631 podStartE2EDuration="2.362970631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:43.362855753 +0000 UTC m=+3.686548603" watchObservedRunningTime="2026-09-16 18:45:43.362970631 +0000 UTC m=+3.686663481" Sep 16 18:45:43.665153 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.665066 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:43.665153 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.665124 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:43.665396 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:43.665304 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:43.665396 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:43.665371 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs podName:650deff9-b845-4ef3-acad-6bdee9667025 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.66535163 +0000 UTC m=+5.989044470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs") pod "network-metrics-daemon-lq68v" (UID: "650deff9-b845-4ef3-acad-6bdee9667025") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:43.665632 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:43.665612 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:43.665712 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:43.665671 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret podName:0fed9f62-c1e7-419f-b5d2-85d3214e9d8e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.665656705 +0000 UTC m=+5.989349545 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret") pod "global-pull-secret-syncer-klhb5" (UID: "0fed9f62-c1e7-419f-b5d2-85d3214e9d8e") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:43.765870 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:43.765813 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qkn\" (UniqueName: \"kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn\") pod \"network-check-target-9k5bx\" (UID: \"94040f40-6ebc-4c73-9280-83b5e973a840\") " pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:43.766068 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:43.766050 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:43.766133 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:43.766078 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:43.766133 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:43.766090 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d5qkn for pod openshift-network-diagnostics/network-check-target-9k5bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:43.766251 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:43.766152 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn podName:94040f40-6ebc-4c73-9280-83b5e973a840 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.766133532 +0000 UTC m=+6.089826379 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qkn" (UniqueName: "kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn") pod "network-check-target-9k5bx" (UID: "94040f40-6ebc-4c73-9280-83b5e973a840") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:44.089608 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:44.089543 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:40:42 +0000 UTC" deadline="2028-06-03 14:13:10.136836446 +0000 UTC" Sep 16 18:45:44.089608 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:44.089606 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15019h27m26.047234333s" Sep 16 18:45:44.332039 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:44.331953 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:44.332195 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:44.332079 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:45:44.332781 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:44.332755 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:44.332894 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:44.332879 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:45:44.367903 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:44.367865 2563 generic.go:356] "Generic (PLEG): container finished" podID="941dab79fa0f2bb2b6d92051cde9534f" containerID="30b7a8c0eef6bd0df5d560d997c0daea4f992484569b23e9d4a564acca7909eb" exitCode=0 Sep 16 18:45:44.368875 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:44.368822 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" event={"ID":"941dab79fa0f2bb2b6d92051cde9534f","Type":"ContainerDied","Data":"30b7a8c0eef6bd0df5d560d997c0daea4f992484569b23e9d4a564acca7909eb"} Sep 16 18:45:45.331384 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:45.330806 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:45.331384 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:45.330975 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:45:45.398772 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:45.398005 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" event={"ID":"941dab79fa0f2bb2b6d92051cde9534f","Type":"ContainerStarted","Data":"117067322b99b0384430bbd9dd85666522d29a55a1f84f8edc0db85c7ec482e6"} Sep 16 18:45:45.684191 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:45.683355 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:45.684191 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:45.683414 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:45.684191 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:45.683561 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:45.684191 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:45.683634 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs podName:650deff9-b845-4ef3-acad-6bdee9667025 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:49.683614216 +0000 UTC m=+10.007307046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs") pod "network-metrics-daemon-lq68v" (UID: "650deff9-b845-4ef3-acad-6bdee9667025") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:45.684191 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:45.684032 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:45.684191 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:45.684083 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret podName:0fed9f62-c1e7-419f-b5d2-85d3214e9d8e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:49.684067341 +0000 UTC m=+10.007760174 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret") pod "global-pull-secret-syncer-klhb5" (UID: "0fed9f62-c1e7-419f-b5d2-85d3214e9d8e") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:45.784668 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:45.784627 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qkn\" (UniqueName: \"kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn\") pod \"network-check-target-9k5bx\" (UID: \"94040f40-6ebc-4c73-9280-83b5e973a840\") " pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:45.784886 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:45.784866 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:45.784966 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:45.784898 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:45.784966 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:45.784914 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d5qkn for pod openshift-network-diagnostics/network-check-target-9k5bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:45.785074 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:45.784981 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn podName:94040f40-6ebc-4c73-9280-83b5e973a840 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:49.7849619 +0000 UTC m=+10.108654733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qkn" (UniqueName: "kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn") pod "network-check-target-9k5bx" (UID: "94040f40-6ebc-4c73-9280-83b5e973a840") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:46.331967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:46.331928 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:46.332458 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:46.332061 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:45:46.332458 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:46.332100 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:46.332458 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:46.332164 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:45:47.331039 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:47.331004 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:47.331243 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:47.331131 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:45:48.331079 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:48.330961 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:48.331079 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:48.330999 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:48.331661 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:48.331101 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:45:48.331661 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:48.331537 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:45:49.331825 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:49.331782 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:49.332319 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:49.331960 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:45:49.722355 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:49.722165 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:49.722355 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:49.722229 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:49.722355 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:49.722345 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:49.722719 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:49.722432 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs podName:650deff9-b845-4ef3-acad-6bdee9667025 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:57.722411315 +0000 UTC m=+18.046104161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs") pod "network-metrics-daemon-lq68v" (UID: "650deff9-b845-4ef3-acad-6bdee9667025") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:49.722719 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:49.722344 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:49.722719 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:49.722506 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret podName:0fed9f62-c1e7-419f-b5d2-85d3214e9d8e nodeName:}" failed. No retries permitted until 2026-09-16 18:45:57.722486474 +0000 UTC m=+18.046179318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret") pod "global-pull-secret-syncer-klhb5" (UID: "0fed9f62-c1e7-419f-b5d2-85d3214e9d8e") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:49.823084 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:49.823044 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qkn\" (UniqueName: \"kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn\") pod \"network-check-target-9k5bx\" (UID: \"94040f40-6ebc-4c73-9280-83b5e973a840\") " pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:49.823300 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:49.823280 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:49.823368 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:49.823311 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:49.823368 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:49.823323 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d5qkn for pod openshift-network-diagnostics/network-check-target-9k5bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:49.823472 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:49.823390 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn podName:94040f40-6ebc-4c73-9280-83b5e973a840 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:57.823369675 +0000 UTC m=+18.147062517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qkn" (UniqueName: "kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn") pod "network-check-target-9k5bx" (UID: "94040f40-6ebc-4c73-9280-83b5e973a840") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:50.332169 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:50.331961 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:50.332169 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:50.332083 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:45:50.334574 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:50.334311 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:50.334574 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:50.334497 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:45:51.331426 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:51.330957 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:51.331426 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:51.331094 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:45:52.331754 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:52.331474 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:52.332213 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:52.331492 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:52.332213 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:52.331900 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:45:52.332213 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:52.331939 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:45:53.331649 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:53.331610 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:53.331863 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:53.331757 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:45:54.331913 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:54.331752 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:54.331913 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:54.331899 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:45:54.332338 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:54.331952 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:54.332338 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:54.332044 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:45:55.330884 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:55.330826 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:55.331062 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:55.330985 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:45:56.331321 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:56.331290 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:56.331678 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:56.331300 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:56.331678 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:56.331386 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:45:56.331678 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:56.331479 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:45:57.330916 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:57.330878 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:57.331174 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:57.331026 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:45:57.786046 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:57.785950 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:57.786493 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:57.786064 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:57.786493 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:57.786114 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:57.786493 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:57.786172 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:57.786493 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:57.786182 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs podName:650deff9-b845-4ef3-acad-6bdee9667025 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.786167202 +0000 UTC m=+34.109860034 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs") pod "network-metrics-daemon-lq68v" (UID: "650deff9-b845-4ef3-acad-6bdee9667025") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:45:57.786493 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:57.786225 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret podName:0fed9f62-c1e7-419f-b5d2-85d3214e9d8e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.786210427 +0000 UTC m=+34.109903271 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret") pod "global-pull-secret-syncer-klhb5" (UID: "0fed9f62-c1e7-419f-b5d2-85d3214e9d8e") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:45:57.887060 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:57.887024 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qkn\" (UniqueName: \"kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn\") pod \"network-check-target-9k5bx\" (UID: \"94040f40-6ebc-4c73-9280-83b5e973a840\") " pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:57.887217 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:57.887161 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:45:57.887217 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:57.887175 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:45:57.887217 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:57.887183 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d5qkn for pod openshift-network-diagnostics/network-check-target-9k5bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:57.887318 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:57.887233 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn podName:94040f40-6ebc-4c73-9280-83b5e973a840 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:13.887221004 +0000 UTC m=+34.210913830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qkn" (UniqueName: "kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn") pod "network-check-target-9k5bx" (UID: "94040f40-6ebc-4c73-9280-83b5e973a840") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:45:58.331931 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:58.331896 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:45:58.332142 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:58.331896 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:45:58.332142 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:58.332046 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:45:58.332142 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:58.332121 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:45:59.331379 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:45:59.331301 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:45:59.331823 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:45:59.331478 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:46:00.332291 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:00.332250 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:00.332707 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:00.332371 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:46:00.332707 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:00.332698 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:00.332807 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:00.332777 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:46:01.331657 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:01.331621 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:46:01.331914 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:01.331732 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:46:02.331387 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.331117 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:02.332286 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.331128 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:02.332286 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:02.331421 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:46:02.332286 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:02.331516 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:46:02.446110 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.446073 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" event={"ID":"4ae005ac-ab99-433a-bd5c-7b8193356ffb","Type":"ContainerStarted","Data":"8b827b1026d9938c1c277432e186c8eab92d22609b858a3744e8418751907f77"} Sep 16 18:46:02.446298 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.446150 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" event={"ID":"4ae005ac-ab99-433a-bd5c-7b8193356ffb","Type":"ContainerStarted","Data":"2f9e0e0141e359247d64d3e7956c4cccd1abbef388e88d26c11ab42a455a30f0"} Sep 16 18:46:02.446298 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.446181 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" event={"ID":"4ae005ac-ab99-433a-bd5c-7b8193356ffb","Type":"ContainerStarted","Data":"d08718db9c24ba85e574ea2d666d35bb3139beef667e6e016bb2e950c58aab29"} Sep 16 18:46:02.446298 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.446217 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" event={"ID":"4ae005ac-ab99-433a-bd5c-7b8193356ffb","Type":"ContainerStarted","Data":"1c295302586dd07afbff16b6e3950ea0ea3129d3061bf08a9341848aebdba17a"} Sep 16 18:46:02.446298 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.446245 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" event={"ID":"4ae005ac-ab99-433a-bd5c-7b8193356ffb","Type":"ContainerStarted","Data":"7e569387ddd2c76106dc528acd224a73c4ee0d0fb893866eb7958eedad5563d5"} Sep 16 18:46:02.446298 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.446276 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" event={"ID":"4ae005ac-ab99-433a-bd5c-7b8193356ffb","Type":"ContainerStarted","Data":"355f363a3a8106ef516f7e3cb9dd90eb0763050befac028cf71da16de3cb1ad9"} Sep 16 18:46:02.447533 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.447504 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jprzs" event={"ID":"97e78433-797a-47b6-9b23-798ddb52e025","Type":"ContainerStarted","Data":"fcbef64b58420b861b714fd6fce218d780d02b6898a4100202ac39ed66e821b2"} Sep 16 18:46:02.449080 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.449039 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h946g" event={"ID":"01d77fab-c94e-46ea-b0e8-1b2b02f5d977","Type":"ContainerStarted","Data":"1aa0190f297eee4df7006911583064fb2b4bc8bf05e00e31734f19a148c487a4"} Sep 16 18:46:02.450793 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.450769 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-m5czp" event={"ID":"f741ffaa-797b-4f41-ac31-6da5fe63d9bd","Type":"ContainerStarted","Data":"35278df30a92848dace0f358a87a3d9fa03d2a77395f233bf2c167caa6bed445"} Sep 16 18:46:02.453599 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.453571 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lh5wh" event={"ID":"f2231582-bd0c-440b-9841-bd4740794c8c","Type":"ContainerStarted","Data":"fa6898230593783367cde377f09ffa38b2e8b5f462e2268c67f59d6e4cfa841a"} Sep 16 18:46:02.455042 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.455021 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" event={"ID":"b2648669-4bd6-4a83-92ef-b694ca40e42c","Type":"ContainerStarted","Data":"eb9a7e9537225f7125256450507b65b003204edaca059d71b829c688f575cedc"} Sep 16 18:46:02.456422 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.456402 2563 generic.go:356] "Generic (PLEG): container finished" podID="003ff6ca-4a01-4b09-98ea-17bf09a51ffe" containerID="6a1affd9bbba19879c5908ca5f65681973620446c5b49545d007c8a8ef01d517" exitCode=0 Sep 16 18:46:02.456512 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.456460 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjvnd" event={"ID":"003ff6ca-4a01-4b09-98ea-17bf09a51ffe","Type":"ContainerDied","Data":"6a1affd9bbba19879c5908ca5f65681973620446c5b49545d007c8a8ef01d517"} Sep 16 18:46:02.458065 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.458046 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" event={"ID":"f038ea97-8f13-448a-a62e-547ae8ba0c67","Type":"ContainerStarted","Data":"a8c8c5a1bcc826960c832338a66ada48ec6c64cf599616837daf52b22f91c62b"} Sep 16 18:46:02.468715 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.468671 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-125.ec2.internal" podStartSLOduration=21.468658284 podStartE2EDuration="21.468658284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:45.416458394 +0000 UTC m=+5.740151245" watchObservedRunningTime="2026-09-16 18:46:02.468658284 +0000 UTC m=+22.792351134" Sep 16 18:46:02.469024 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.469003 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jprzs" podStartSLOduration=4.29857915 podStartE2EDuration="22.468997006s" totalImagesPullingTime="18.170417856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:02.468409291 +0000 UTC m=+22.792102164" watchObservedRunningTime="2026-09-16 18:46:02.468997006 +0000 UTC m=+22.792689854" Sep 16 18:46:02.518426 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.518374 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-m5czp" podStartSLOduration=4.410657742 podStartE2EDuration="22.518357633s" totalImagesPullingTime="18.107699891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:02.517812982 +0000 UTC m=+22.841505843" watchObservedRunningTime="2026-09-16 18:46:02.518357633 +0000 UTC m=+22.842050482" Sep 16 18:46:02.536955 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.536763 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rjhmn" podStartSLOduration=4.423388839 podStartE2EDuration="22.536748026s" totalImagesPullingTime="18.113359187s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:02.53650617 +0000 UTC m=+22.860199011" watchObservedRunningTime="2026-09-16 18:46:02.536748026 +0000 UTC m=+22.860440875" Sep 16 18:46:02.556806 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.556758 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lh5wh" podStartSLOduration=4.44639416 podStartE2EDuration="22.556743023s" totalImagesPullingTime="18.110348863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:02.556712542 +0000 UTC m=+22.880405391" watchObservedRunningTime="2026-09-16 18:46:02.556743023 +0000 UTC m=+22.880435872" Sep 16 18:46:02.578275 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.578248 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 18:46:02.582934 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:02.582843 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h946g" podStartSLOduration=4.478397954 podStartE2EDuration="22.582814272s" totalImagesPullingTime="18.104416318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:02.582415352 +0000 UTC m=+22.906108193" watchObservedRunningTime="2026-09-16 18:46:02.582814272 +0000 UTC m=+22.906507121" Sep 16 18:46:03.108012 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:03.107887 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:46:02.578276581Z","UUID":"ddb7636b-015e-400a-a561-43e5270db7c1","Handler":null,"Name":"","Endpoint":""} Sep 16 18:46:03.109569 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:03.109539 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 18:46:03.109705 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:03.109578 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 18:46:03.331379 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:03.331346 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:46:03.331560 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:03.331498 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:46:03.463872 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:03.463811 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" event={"ID":"f038ea97-8f13-448a-a62e-547ae8ba0c67","Type":"ContainerStarted","Data":"7d4a53702a6b10c7e1dcb4c69c5bda77cbb77bf046b05b13af7aac5f782b67d8"} Sep 16 18:46:03.466249 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:03.466217 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5qjc7" event={"ID":"8a70859e-eddb-45d0-b551-7c79611c1d90","Type":"ContainerStarted","Data":"b02f57a779f4cdfc1c41a56a4948ea76b89d8ec5d117d31270315b3cc6e14830"} Sep 16 18:46:03.485586 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:03.485535 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5qjc7" podStartSLOduration=5.384330465 podStartE2EDuration="23.485520134s" totalImagesPullingTime="18.101189669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:03.485365374 +0000 UTC m=+23.809058223" watchObservedRunningTime="2026-09-16 18:46:03.485520134 +0000 UTC m=+23.809212980" Sep 16 18:46:04.331202 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:04.331117 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:04.331202 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:04.331147 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:04.331380 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:04.331242 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:46:04.331380 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:04.331357 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:46:04.471270 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:04.471227 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" event={"ID":"f038ea97-8f13-448a-a62e-547ae8ba0c67","Type":"ContainerStarted","Data":"9c9dd51c1ad03cd6067a77938077fec1c4b60c39169fa471bf661e2bcf4e2ece"} Sep 16 18:46:04.475802 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:04.475316 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" event={"ID":"4ae005ac-ab99-433a-bd5c-7b8193356ffb","Type":"ContainerStarted","Data":"3e8709c56c5540646d0cc932e046433954ef0d624e9e376e1b987089dec10cf1"} Sep 16 18:46:04.495768 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:04.495708 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sgwmk" podStartSLOduration=4.6646321870000005 podStartE2EDuration="24.49568549s" totalImagesPullingTime="19.831053303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:04.495039023 +0000 UTC m=+24.818731873" watchObservedRunningTime="2026-09-16 18:46:04.49568549 +0000 UTC m=+24.819378340" Sep 16 18:46:05.331501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:05.331459 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:46:05.331724 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:05.331625 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:46:06.331114 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:06.331075 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:06.331546 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:06.331213 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:46:06.331546 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:06.331083 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:06.331546 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:06.331367 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:46:06.877746 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:06.877459 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:46:06.878413 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:06.878141 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:46:07.331409 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:07.331315 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:46:07.331966 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:07.331429 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:46:07.484300 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:07.484265 2563 generic.go:356] "Generic (PLEG): container finished" podID="003ff6ca-4a01-4b09-98ea-17bf09a51ffe" containerID="ee163f457436eaa43f9fd0a855f7052b3510b7951e4e07481f38c15ab604dca3" exitCode=0 Sep 16 18:46:07.484496 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:07.484347 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjvnd" event={"ID":"003ff6ca-4a01-4b09-98ea-17bf09a51ffe","Type":"ContainerDied","Data":"ee163f457436eaa43f9fd0a855f7052b3510b7951e4e07481f38c15ab604dca3"} Sep 16 18:46:07.488531 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:07.488502 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" event={"ID":"4ae005ac-ab99-433a-bd5c-7b8193356ffb","Type":"ContainerStarted","Data":"119488e20a489f0519d97bc0e446560b369fe34fa12f11a9d0397023bd95884a"} Sep 16 18:46:07.488818 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:07.488790 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:46:07.488935 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:07.488895 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:46:07.504541 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:07.504515 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:46:07.536851 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:07.536782 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" podStartSLOduration=9.342260425 podStartE2EDuration="27.536763916s" totalImagesPullingTime="18.194503491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:07.535557502 +0000 UTC m=+27.859250361" watchObservedRunningTime="2026-09-16 18:46:07.536763916 +0000 UTC m=+27.860456765" Sep 16 18:46:08.331554 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:08.331514 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:08.332238 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:08.331530 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:08.332238 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:08.331668 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:46:08.332238 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:08.331740 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:46:08.492574 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:08.492427 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:46:08.508470 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:08.508411 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:46:08.703264 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:08.703232 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9k5bx"] Sep 16 18:46:08.703435 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:08.703382 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:08.703519 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:08.703494 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:46:08.706188 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:08.706154 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-klhb5"] Sep 16 18:46:08.706346 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:08.706294 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:08.706430 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:08.706405 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:46:08.706750 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:08.706727 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lq68v"] Sep 16 18:46:08.706853 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:08.706826 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:46:08.706952 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:08.706932 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:46:09.495954 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:09.495914 2563 generic.go:356] "Generic (PLEG): container finished" podID="003ff6ca-4a01-4b09-98ea-17bf09a51ffe" containerID="74bda920f8c2c84e96cfae600d31d0e89ac676f3bac0779b7ac4cf3c45b54c20" exitCode=0 Sep 16 18:46:09.496339 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:09.495967 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjvnd" event={"ID":"003ff6ca-4a01-4b09-98ea-17bf09a51ffe","Type":"ContainerDied","Data":"74bda920f8c2c84e96cfae600d31d0e89ac676f3bac0779b7ac4cf3c45b54c20"} Sep 16 18:46:09.982594 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:09.982454 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:46:09.982750 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:09.982600 2563 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 16 18:46:09.983425 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:09.983400 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-m5czp" Sep 16 18:46:10.332721 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:10.332691 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:10.332894 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:10.332725 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:46:10.332894 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:10.332812 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:10.332894 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:10.332825 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:46:10.333058 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:10.332921 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:46:10.333058 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:10.333021 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:46:10.501440 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:10.500769 2563 generic.go:356] "Generic (PLEG): container finished" podID="003ff6ca-4a01-4b09-98ea-17bf09a51ffe" containerID="da04baf2bf7ec08a02c8aeeda564dcb61f47bb0d317a89831a09ee1ab1e45903" exitCode=0 Sep 16 18:46:10.501440 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:10.501148 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjvnd" event={"ID":"003ff6ca-4a01-4b09-98ea-17bf09a51ffe","Type":"ContainerDied","Data":"da04baf2bf7ec08a02c8aeeda564dcb61f47bb0d317a89831a09ee1ab1e45903"} Sep 16 18:46:12.331604 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:12.331230 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:12.331604 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:12.331261 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:46:12.331604 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:12.331371 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:46:12.331604 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:12.331387 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:12.331604 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:12.331482 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:46:12.331604 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:12.331563 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:46:13.807808 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:13.807589 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:46:13.808402 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:13.807760 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:46:13.808402 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:13.807910 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:13.808402 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:13.807935 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs podName:650deff9-b845-4ef3-acad-6bdee9667025 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:45.807912252 +0000 UTC m=+66.131605094 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs") pod "network-metrics-daemon-lq68v" (UID: "650deff9-b845-4ef3-acad-6bdee9667025") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 18:46:13.808402 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:13.807996 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 18:46:13.808402 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:13.808049 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret podName:0fed9f62-c1e7-419f-b5d2-85d3214e9d8e nodeName:}" failed. No retries permitted until 2026-09-16 18:46:45.808036208 +0000 UTC m=+66.131729039 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret") pod "global-pull-secret-syncer-klhb5" (UID: "0fed9f62-c1e7-419f-b5d2-85d3214e9d8e") : object "kube-system"/"original-pull-secret" not registered Sep 16 18:46:13.908484 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:13.908443 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qkn\" (UniqueName: \"kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn\") pod \"network-check-target-9k5bx\" (UID: \"94040f40-6ebc-4c73-9280-83b5e973a840\") " pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:13.908672 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:13.908647 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 18:46:13.908736 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:13.908681 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 18:46:13.908736 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:13.908694 2563 projected.go:196] Error preparing data for projected volume kube-api-access-d5qkn for pod openshift-network-diagnostics/network-check-target-9k5bx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:46:13.908849 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:13.908761 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn podName:94040f40-6ebc-4c73-9280-83b5e973a840 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:45.908743308 +0000 UTC m=+66.232436152 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-d5qkn" (UniqueName: "kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn") pod "network-check-target-9k5bx" (UID: "94040f40-6ebc-4c73-9280-83b5e973a840") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 18:46:14.331411 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.331379 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:14.331587 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.331419 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:46:14.331587 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.331383 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:14.331587 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:14.331508 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-klhb5" podUID="0fed9f62-c1e7-419f-b5d2-85d3214e9d8e" Sep 16 18:46:14.331750 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:14.331580 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9k5bx" podUID="94040f40-6ebc-4c73-9280-83b5e973a840" Sep 16 18:46:14.331750 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:14.331668 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:46:14.427167 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.427130 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-125.ec2.internal" event="NodeReady" Sep 16 18:46:14.427348 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.427292 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 18:46:14.491002 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.490920 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xrfb7"] Sep 16 18:46:14.522987 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.522955 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8vrw2"] Sep 16 18:46:14.523154 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.523094 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.525137 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.525113 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 18:46:14.525280 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.525255 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 18:46:14.525397 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.525376 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgt44\"" Sep 16 18:46:14.539134 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.539001 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xrfb7"] Sep 16 18:46:14.539134 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.539083 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8vrw2" Sep 16 18:46:14.539134 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.539100 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8vrw2"] Sep 16 18:46:14.541533 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.541319 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z45rg\"" Sep 16 18:46:14.541533 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.541382 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 18:46:14.541533 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.541408 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 18:46:14.542166 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.542014 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 18:46:14.614934 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.614893 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ceca87-379c-489d-b13f-6365d0a612b2-tmp-dir\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.614934 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.614936 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1bc280d-ee46-48fe-a939-7b6d82bb1c92-cert\") pod \"ingress-canary-8vrw2\" (UID: \"d1bc280d-ee46-48fe-a939-7b6d82bb1c92\") " pod="openshift-ingress-canary/ingress-canary-8vrw2" Sep 16 18:46:14.615184 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.615027 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ceca87-379c-489d-b13f-6365d0a612b2-config-volume\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.615184 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.615092 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsz94\" (UniqueName: \"kubernetes.io/projected/d1bc280d-ee46-48fe-a939-7b6d82bb1c92-kube-api-access-gsz94\") pod \"ingress-canary-8vrw2\" (UID: \"d1bc280d-ee46-48fe-a939-7b6d82bb1c92\") " pod="openshift-ingress-canary/ingress-canary-8vrw2" Sep 16 18:46:14.615184 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.615115 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.615184 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.615132 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plbq4\" (UniqueName: \"kubernetes.io/projected/b1ceca87-379c-489d-b13f-6365d0a612b2-kube-api-access-plbq4\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.715594 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.715556 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsz94\" (UniqueName: \"kubernetes.io/projected/d1bc280d-ee46-48fe-a939-7b6d82bb1c92-kube-api-access-gsz94\") pod \"ingress-canary-8vrw2\" (UID: \"d1bc280d-ee46-48fe-a939-7b6d82bb1c92\") " pod="openshift-ingress-canary/ingress-canary-8vrw2" Sep 16 18:46:14.715594 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.715590 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.715850 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.715607 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbq4\" (UniqueName: \"kubernetes.io/projected/b1ceca87-379c-489d-b13f-6365d0a612b2-kube-api-access-plbq4\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.715850 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:14.715729 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:14.715850 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:14.715809 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls podName:b1ceca87-379c-489d-b13f-6365d0a612b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:15.215787955 +0000 UTC m=+35.539480782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls") pod "dns-default-xrfb7" (UID: "b1ceca87-379c-489d-b13f-6365d0a612b2") : secret "dns-default-metrics-tls" not found Sep 16 18:46:14.715992 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.715868 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ceca87-379c-489d-b13f-6365d0a612b2-tmp-dir\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.715992 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.715914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1bc280d-ee46-48fe-a939-7b6d82bb1c92-cert\") pod \"ingress-canary-8vrw2\" (UID: \"d1bc280d-ee46-48fe-a939-7b6d82bb1c92\") " pod="openshift-ingress-canary/ingress-canary-8vrw2" Sep 16 18:46:14.715992 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.715971 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ceca87-379c-489d-b13f-6365d0a612b2-config-volume\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.716190 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.716173 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1ceca87-379c-489d-b13f-6365d0a612b2-tmp-dir\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.716449 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.716431 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ceca87-379c-489d-b13f-6365d0a612b2-config-volume\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.720122 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.720101 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d1bc280d-ee46-48fe-a939-7b6d82bb1c92-cert\") pod \"ingress-canary-8vrw2\" (UID: \"d1bc280d-ee46-48fe-a939-7b6d82bb1c92\") " pod="openshift-ingress-canary/ingress-canary-8vrw2" Sep 16 18:46:14.725648 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.725627 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbq4\" (UniqueName: \"kubernetes.io/projected/b1ceca87-379c-489d-b13f-6365d0a612b2-kube-api-access-plbq4\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:14.725871 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.725855 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsz94\" (UniqueName: \"kubernetes.io/projected/d1bc280d-ee46-48fe-a939-7b6d82bb1c92-kube-api-access-gsz94\") pod \"ingress-canary-8vrw2\" (UID: \"d1bc280d-ee46-48fe-a939-7b6d82bb1c92\") " pod="openshift-ingress-canary/ingress-canary-8vrw2" Sep 16 18:46:14.849666 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:14.849636 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8vrw2" Sep 16 18:46:15.220218 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:15.220180 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:15.220353 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:15.220319 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:15.220405 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:15.220389 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls podName:b1ceca87-379c-489d-b13f-6365d0a612b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:16.220374692 +0000 UTC m=+36.544067523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls") pod "dns-default-xrfb7" (UID: "b1ceca87-379c-489d-b13f-6365d0a612b2") : secret "dns-default-metrics-tls" not found Sep 16 18:46:15.282311 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:15.282088 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8vrw2"] Sep 16 18:46:15.296844 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:46:15.296800 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1bc280d_ee46_48fe_a939_7b6d82bb1c92.slice/crio-1fad6ba8215a3d967005653d461eaf4d736f5a0c47f8602f605dcbb0f8340d62 WatchSource:0}: Error finding container 1fad6ba8215a3d967005653d461eaf4d736f5a0c47f8602f605dcbb0f8340d62: Status 404 returned error can't find the container with id 1fad6ba8215a3d967005653d461eaf4d736f5a0c47f8602f605dcbb0f8340d62 Sep 16 18:46:15.516056 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:15.516019 2563 generic.go:356] "Generic (PLEG): container finished" podID="003ff6ca-4a01-4b09-98ea-17bf09a51ffe" containerID="d20935329c9be5ea0d5f36dece8addfdbff67fcfa10a4b2cc22ab232426973dc" exitCode=0 Sep 16 18:46:15.516244 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:15.516110 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjvnd" event={"ID":"003ff6ca-4a01-4b09-98ea-17bf09a51ffe","Type":"ContainerDied","Data":"d20935329c9be5ea0d5f36dece8addfdbff67fcfa10a4b2cc22ab232426973dc"} Sep 16 18:46:15.517193 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:15.517172 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8vrw2" event={"ID":"d1bc280d-ee46-48fe-a939-7b6d82bb1c92","Type":"ContainerStarted","Data":"1fad6ba8215a3d967005653d461eaf4d736f5a0c47f8602f605dcbb0f8340d62"} Sep 16 18:46:16.226802 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.226762 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:16.227518 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:16.226924 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:16.227518 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:16.226985 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls podName:b1ceca87-379c-489d-b13f-6365d0a612b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:18.226968548 +0000 UTC m=+38.550661385 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls") pod "dns-default-xrfb7" (UID: "b1ceca87-379c-489d-b13f-6365d0a612b2") : secret "dns-default-metrics-tls" not found Sep 16 18:46:16.335721 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.335689 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:16.335932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.335700 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:16.336062 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.335826 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:46:16.338266 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.338247 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:46:16.338931 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.338895 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:46:16.339290 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.339264 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmvxr\"" Sep 16 18:46:16.339584 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.339564 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:46:16.339806 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.339790 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7n2q\"" Sep 16 18:46:16.342627 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.339203 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:46:16.523878 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.523823 2563 generic.go:356] "Generic (PLEG): container finished" podID="003ff6ca-4a01-4b09-98ea-17bf09a51ffe" containerID="2a1639c9521c4b3820cbdb1bb07fa5e6fff5b8bc11b0025b0ff3c2e3499be3c3" exitCode=0 Sep 16 18:46:16.524068 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:16.523920 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjvnd" event={"ID":"003ff6ca-4a01-4b09-98ea-17bf09a51ffe","Type":"ContainerDied","Data":"2a1639c9521c4b3820cbdb1bb07fa5e6fff5b8bc11b0025b0ff3c2e3499be3c3"} Sep 16 18:46:17.530206 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:17.530172 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vjvnd" event={"ID":"003ff6ca-4a01-4b09-98ea-17bf09a51ffe","Type":"ContainerStarted","Data":"737f9339201af51e6dce427be86a3c07b73f39b110979dafa032b24ca5492c59"} Sep 16 18:46:17.558791 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:17.558734 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vjvnd" podStartSLOduration=9.095993945 podStartE2EDuration="37.558715331s" totalImagesPullingTime="28.462721386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:17.558258529 +0000 UTC m=+37.881951507" watchObservedRunningTime="2026-09-16 18:46:17.558715331 +0000 UTC m=+37.882408174" Sep 16 18:46:18.243116 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:18.243089 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:18.243264 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:18.243244 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:18.243324 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:18.243314 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls podName:b1ceca87-379c-489d-b13f-6365d0a612b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:22.243297173 +0000 UTC m=+42.566990005 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls") pod "dns-default-xrfb7" (UID: "b1ceca87-379c-489d-b13f-6365d0a612b2") : secret "dns-default-metrics-tls" not found Sep 16 18:46:18.534270 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:18.534230 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8vrw2" event={"ID":"d1bc280d-ee46-48fe-a939-7b6d82bb1c92","Type":"ContainerStarted","Data":"e6ecbb6481039e4adca49a9058014a076aae23b665397047e1e26cd3d0cbb469"} Sep 16 18:46:22.269123 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:22.269078 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:22.269520 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:22.269245 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:22.269520 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:22.269317 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls podName:b1ceca87-379c-489d-b13f-6365d0a612b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:30.269300066 +0000 UTC m=+50.592992900 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls") pod "dns-default-xrfb7" (UID: "b1ceca87-379c-489d-b13f-6365d0a612b2") : secret "dns-default-metrics-tls" not found Sep 16 18:46:30.326997 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:30.326955 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:30.327435 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:30.327095 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:30.327435 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:30.327164 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls podName:b1ceca87-379c-489d-b13f-6365d0a612b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:46.327142512 +0000 UTC m=+66.650835353 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls") pod "dns-default-xrfb7" (UID: "b1ceca87-379c-489d-b13f-6365d0a612b2") : secret "dns-default-metrics-tls" not found Sep 16 18:46:40.512087 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:40.512059 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bx8nl" Sep 16 18:46:40.542879 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:40.542803 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8vrw2" podStartSLOduration=24.197898279 podStartE2EDuration="26.542787615s" totalImagesPullingTime="2.344889336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:18.558739617 +0000 UTC m=+38.882432465" watchObservedRunningTime="2026-09-16 18:46:40.542787615 +0000 UTC m=+60.866480462" Sep 16 18:46:45.841292 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:45.841244 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:45.841718 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:45.841303 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:46:45.843770 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:45.843750 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 18:46:45.843842 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:45.843805 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 18:46:45.851486 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:45.851462 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 18:46:45.851594 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:45.851541 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs podName:650deff9-b845-4ef3-acad-6bdee9667025 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:49.851521056 +0000 UTC m=+130.175213891 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs") pod "network-metrics-daemon-lq68v" (UID: "650deff9-b845-4ef3-acad-6bdee9667025") : secret "metrics-daemon-secret" not found Sep 16 18:46:45.854827 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:45.854807 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0fed9f62-c1e7-419f-b5d2-85d3214e9d8e-original-pull-secret\") pod \"global-pull-secret-syncer-klhb5\" (UID: \"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e\") " pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:45.941767 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:45.941730 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5qkn\" (UniqueName: \"kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn\") pod \"network-check-target-9k5bx\" (UID: \"94040f40-6ebc-4c73-9280-83b5e973a840\") " pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:45.944284 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:45.944261 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 18:46:45.954087 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:45.954062 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 18:46:45.964983 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:45.964955 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5qkn\" (UniqueName: \"kubernetes.io/projected/94040f40-6ebc-4c73-9280-83b5e973a840-kube-api-access-d5qkn\") pod \"network-check-target-9k5bx\" (UID: \"94040f40-6ebc-4c73-9280-83b5e973a840\") " pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:46.050021 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:46.049988 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-klhb5" Sep 16 18:46:46.061502 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:46.061478 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hmvxr\"" Sep 16 18:46:46.069905 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:46.069866 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:46.344747 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:46.344717 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:46:46.344920 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:46.344881 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:46:46.344964 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:46:46.344943 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls podName:b1ceca87-379c-489d-b13f-6365d0a612b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:18.344929131 +0000 UTC m=+98.668621958 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls") pod "dns-default-xrfb7" (UID: "b1ceca87-379c-489d-b13f-6365d0a612b2") : secret "dns-default-metrics-tls" not found Sep 16 18:46:46.353290 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:46.353213 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-klhb5"] Sep 16 18:46:46.357143 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:46:46.357103 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fed9f62_c1e7_419f_b5d2_85d3214e9d8e.slice/crio-f43ccb595052d235efa1db6bef16ce27b212171c105e2ddf97055fb15f58a9b9 WatchSource:0}: Error finding container f43ccb595052d235efa1db6bef16ce27b212171c105e2ddf97055fb15f58a9b9: Status 404 returned error can't find the container with id f43ccb595052d235efa1db6bef16ce27b212171c105e2ddf97055fb15f58a9b9 Sep 16 18:46:46.373236 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:46.373168 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9k5bx"] Sep 16 18:46:46.378287 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:46:46.378253 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94040f40_6ebc_4c73_9280_83b5e973a840.slice/crio-1732859df20bcbfb4311b28b0b89ef9ed6d0f59344e21e89a3e7067130ee5ecc WatchSource:0}: Error finding container 1732859df20bcbfb4311b28b0b89ef9ed6d0f59344e21e89a3e7067130ee5ecc: Status 404 returned error can't find the container with id 1732859df20bcbfb4311b28b0b89ef9ed6d0f59344e21e89a3e7067130ee5ecc Sep 16 18:46:46.602904 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:46.602866 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9k5bx" event={"ID":"94040f40-6ebc-4c73-9280-83b5e973a840","Type":"ContainerStarted","Data":"1732859df20bcbfb4311b28b0b89ef9ed6d0f59344e21e89a3e7067130ee5ecc"} Sep 16 18:46:46.603953 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:46.603889 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-klhb5" event={"ID":"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e","Type":"ContainerStarted","Data":"f43ccb595052d235efa1db6bef16ce27b212171c105e2ddf97055fb15f58a9b9"} Sep 16 18:46:51.618229 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:51.618158 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-klhb5" event={"ID":"0fed9f62-c1e7-419f-b5d2-85d3214e9d8e","Type":"ContainerStarted","Data":"b132d1d6669acf405f536ea91afbdeb15d7ac824efd6b84a9c70543b65c44a54"} Sep 16 18:46:51.619631 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:51.619603 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9k5bx" event={"ID":"94040f40-6ebc-4c73-9280-83b5e973a840","Type":"ContainerStarted","Data":"1e9f1c4193bfc9778a98c893ff4bf4ffdafaeedb07df783839819bd8ea1b0b83"} Sep 16 18:46:51.619778 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:51.619760 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:46:51.634875 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:51.634810 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-klhb5" podStartSLOduration=67.336888518 podStartE2EDuration="1m11.634795084s" totalImagesPullingTime="4.297906566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:51.634154351 +0000 UTC m=+71.957847225" watchObservedRunningTime="2026-09-16 18:46:51.634795084 +0000 UTC m=+71.958487932" Sep 16 18:46:51.649978 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:46:51.649920 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9k5bx" podStartSLOduration=67.378930751 podStartE2EDuration="1m11.649901582s" totalImagesPullingTime="4.270970831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:51.649314727 +0000 UTC m=+71.973007578" watchObservedRunningTime="2026-09-16 18:46:51.649901582 +0000 UTC m=+71.973594431" Sep 16 18:47:18.366526 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:18.366483 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:47:18.367031 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:18.366662 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 18:47:18.367031 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:18.366764 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls podName:b1ceca87-379c-489d-b13f-6365d0a612b2 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:22.366741975 +0000 UTC m=+162.690434822 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls") pod "dns-default-xrfb7" (UID: "b1ceca87-379c-489d-b13f-6365d0a612b2") : secret "dns-default-metrics-tls" not found Sep 16 18:47:22.624433 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:22.624395 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9k5bx" Sep 16 18:47:32.967405 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:32.967363 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p"] Sep 16 18:47:32.969286 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:32.969269 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:32.971299 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:32.971275 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 16 18:47:32.972826 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:32.972799 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 18:47:32.972957 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:32.972823 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 18:47:32.972957 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:32.972823 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 16 18:47:32.974962 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:32.974942 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qm9dd\"" Sep 16 18:47:32.997626 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:32.997588 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p"] Sep 16 18:47:33.057995 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.057962 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl"] Sep 16 18:47:33.059751 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.059735 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" Sep 16 18:47:33.063345 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.063319 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-gk7mq\"" Sep 16 18:47:33.063345 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.063321 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 16 18:47:33.063525 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.063472 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 16 18:47:33.063608 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.063594 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 16 18:47:33.063906 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.063892 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:47:33.067930 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.067907 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bmrb\" (UniqueName: \"kubernetes.io/projected/4f1e811d-0e94-42b6-888c-3e2f1952434c-kube-api-access-6bmrb\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:33.068053 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.067939 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4f1e811d-0e94-42b6-888c-3e2f1952434c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:33.068053 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.067981 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:33.074300 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.074272 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-95c96d4d4-gg962"] Sep 16 18:47:33.079337 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.079245 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-f2hw9"] Sep 16 18:47:33.079590 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.079568 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.081586 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.081563 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl"] Sep 16 18:47:33.081732 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.081712 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.082082 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.082056 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 16 18:47:33.082229 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.082212 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 16 18:47:33.082322 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.082307 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 18:47:33.082374 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.082313 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 16 18:47:33.082374 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.082337 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lr9kh\"" Sep 16 18:47:33.082571 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.082558 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 18:47:33.085187 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.085169 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 16 18:47:33.085292 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.085228 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 16 18:47:33.085863 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.085847 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-twhzz\"" Sep 16 18:47:33.085945 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.085861 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 16 18:47:33.086213 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.086197 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 16 18:47:33.088341 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.088323 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 16 18:47:33.089801 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.089782 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-f2hw9"] Sep 16 18:47:33.094564 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.094541 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-95c96d4d4-gg962"] Sep 16 18:47:33.104491 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.104464 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 16 18:47:33.168411 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168371 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bmrb\" (UniqueName: \"kubernetes.io/projected/4f1e811d-0e94-42b6-888c-3e2f1952434c-kube-api-access-6bmrb\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:33.168411 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168409 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj92c\" (UniqueName: \"kubernetes.io/projected/2a1abe11-738f-42b1-be3c-fbd279e38659-kube-api-access-pj92c\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.168627 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168431 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/322e4d63-e09c-4b27-8681-32b11e13c0cf-trusted-ca\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.168627 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168452 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg26k\" (UniqueName: \"kubernetes.io/projected/322e4d63-e09c-4b27-8681-32b11e13c0cf-kube-api-access-dg26k\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.168627 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168519 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4f1e811d-0e94-42b6-888c-3e2f1952434c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:33.168627 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168558 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-default-certificate\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.168627 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168611 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-stats-auth\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.168821 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168639 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/322e4d63-e09c-4b27-8681-32b11e13c0cf-config\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.168821 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168675 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:33.168821 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168697 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.168821 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168717 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/322e4d63-e09c-4b27-8681-32b11e13c0cf-serving-cert\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.168821 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168752 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/492c472f-5f87-465e-9534-058f606e5944-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4rpwl\" (UID: \"492c472f-5f87-465e-9534-058f606e5944\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" Sep 16 18:47:33.168821 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.168784 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:33.169102 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168881 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/492c472f-5f87-465e-9534-058f606e5944-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4rpwl\" (UID: \"492c472f-5f87-465e-9534-058f606e5944\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" Sep 16 18:47:33.169102 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168927 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtcn\" (UniqueName: \"kubernetes.io/projected/492c472f-5f87-465e-9534-058f606e5944-kube-api-access-4dtcn\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4rpwl\" (UID: \"492c472f-5f87-465e-9534-058f606e5944\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" Sep 16 18:47:33.169102 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.168944 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls podName:4f1e811d-0e94-42b6-888c-3e2f1952434c nodeName:}" failed. No retries permitted until 2026-09-16 18:47:33.668927215 +0000 UTC m=+113.992620047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-hms5p" (UID: "4f1e811d-0e94-42b6-888c-3e2f1952434c") : secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:33.169102 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.168977 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.169238 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.169204 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/4f1e811d-0e94-42b6-888c-3e2f1952434c-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:33.187126 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.187092 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv"] Sep 16 18:47:33.189089 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.189073 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:47:33.202294 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.202272 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-z6hvf\"" Sep 16 18:47:33.202418 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.202295 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 16 18:47:33.211898 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.211874 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 16 18:47:33.214293 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.214272 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bmrb\" (UniqueName: \"kubernetes.io/projected/4f1e811d-0e94-42b6-888c-3e2f1952434c-kube-api-access-6bmrb\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:33.223793 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.223627 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv"] Sep 16 18:47:33.270065 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-default-certificate\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.270065 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270080 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-stats-auth\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.270338 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270100 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/322e4d63-e09c-4b27-8681-32b11e13c0cf-config\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.270338 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270123 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:47:33.270338 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270224 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.270338 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270276 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/322e4d63-e09c-4b27-8681-32b11e13c0cf-serving-cert\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.270338 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270325 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/492c472f-5f87-465e-9534-058f606e5944-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4rpwl\" (UID: \"492c472f-5f87-465e-9534-058f606e5944\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" Sep 16 18:47:33.270649 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270363 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/492c472f-5f87-465e-9534-058f606e5944-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4rpwl\" (UID: \"492c472f-5f87-465e-9534-058f606e5944\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" Sep 16 18:47:33.270649 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.270407 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle podName:2a1abe11-738f-42b1-be3c-fbd279e38659 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:33.77038774 +0000 UTC m=+114.094080579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle") pod "router-default-95c96d4d4-gg962" (UID: "2a1abe11-738f-42b1-be3c-fbd279e38659") : configmap references non-existent config key: service-ca.crt Sep 16 18:47:33.270755 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270681 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtcn\" (UniqueName: \"kubernetes.io/projected/492c472f-5f87-465e-9534-058f606e5944-kube-api-access-4dtcn\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4rpwl\" (UID: \"492c472f-5f87-465e-9534-058f606e5944\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" Sep 16 18:47:33.270755 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270742 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.270876 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270791 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj92c\" (UniqueName: \"kubernetes.io/projected/2a1abe11-738f-42b1-be3c-fbd279e38659-kube-api-access-pj92c\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.270876 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270816 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/322e4d63-e09c-4b27-8681-32b11e13c0cf-trusted-ca\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.270876 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270858 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg26k\" (UniqueName: \"kubernetes.io/projected/322e4d63-e09c-4b27-8681-32b11e13c0cf-kube-api-access-dg26k\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.271022 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270898 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:47:33.271022 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.270937 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/322e4d63-e09c-4b27-8681-32b11e13c0cf-config\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.271022 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.271012 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/492c472f-5f87-465e-9534-058f606e5944-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4rpwl\" (UID: \"492c472f-5f87-465e-9534-058f606e5944\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" Sep 16 18:47:33.271149 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.271033 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 18:47:33.271149 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.271095 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs podName:2a1abe11-738f-42b1-be3c-fbd279e38659 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:33.7710764 +0000 UTC m=+114.094769227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs") pod "router-default-95c96d4d4-gg962" (UID: "2a1abe11-738f-42b1-be3c-fbd279e38659") : secret "router-metrics-certs-default" not found Sep 16 18:47:33.272761 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.272732 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/322e4d63-e09c-4b27-8681-32b11e13c0cf-trusted-ca\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.272962 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.272944 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/322e4d63-e09c-4b27-8681-32b11e13c0cf-serving-cert\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.273144 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.273114 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-stats-auth\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.273261 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.273244 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/492c472f-5f87-465e-9534-058f606e5944-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4rpwl\" (UID: \"492c472f-5f87-465e-9534-058f606e5944\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" Sep 16 18:47:33.273369 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.273351 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-default-certificate\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.287724 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.287696 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtcn\" (UniqueName: \"kubernetes.io/projected/492c472f-5f87-465e-9534-058f606e5944-kube-api-access-4dtcn\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4rpwl\" (UID: \"492c472f-5f87-465e-9534-058f606e5944\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" Sep 16 18:47:33.288510 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.288488 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj92c\" (UniqueName: \"kubernetes.io/projected/2a1abe11-738f-42b1-be3c-fbd279e38659-kube-api-access-pj92c\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.290300 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.290279 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg26k\" (UniqueName: \"kubernetes.io/projected/322e4d63-e09c-4b27-8681-32b11e13c0cf-kube-api-access-dg26k\") pod \"console-operator-696884fb9c-f2hw9\" (UID: \"322e4d63-e09c-4b27-8681-32b11e13c0cf\") " pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.368139 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.368100 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" Sep 16 18:47:33.371753 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.371734 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:47:33.371847 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.371775 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:47:33.371958 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.371942 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 18:47:33.372012 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.372000 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert podName:7b5367e2-4fb0-43ec-8046-f1fdf490abeb nodeName:}" failed. No retries permitted until 2026-09-16 18:47:33.871984619 +0000 UTC m=+114.195677451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqzzv" (UID: "7b5367e2-4fb0-43ec-8046-f1fdf490abeb") : secret "networking-console-plugin-cert" not found Sep 16 18:47:33.372471 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.372442 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:47:33.396700 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.396664 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:33.673741 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.673709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:33.673921 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.673878 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:33.673964 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.673946 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls podName:4f1e811d-0e94-42b6-888c-3e2f1952434c nodeName:}" failed. No retries permitted until 2026-09-16 18:47:34.673930192 +0000 UTC m=+114.997623019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-hms5p" (UID: "4f1e811d-0e94-42b6-888c-3e2f1952434c") : secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:33.682184 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.682128 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl"] Sep 16 18:47:33.686016 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:47:33.685992 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492c472f_5f87_465e_9534_058f606e5944.slice/crio-6fc8fa084af3a6ac8bfcd3d5ed1ddd2b4239b703ed1aed1e6dca12cae00c613f WatchSource:0}: Error finding container 6fc8fa084af3a6ac8bfcd3d5ed1ddd2b4239b703ed1aed1e6dca12cae00c613f: Status 404 returned error can't find the container with id 6fc8fa084af3a6ac8bfcd3d5ed1ddd2b4239b703ed1aed1e6dca12cae00c613f Sep 16 18:47:33.704560 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.704533 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-f2hw9"] Sep 16 18:47:33.708546 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:47:33.708523 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322e4d63_e09c_4b27_8681_32b11e13c0cf.slice/crio-f9999745b78b5ae8f0196855f747fe2c0856d24b02ca3a5de1b0e633d3998765 WatchSource:0}: Error finding container f9999745b78b5ae8f0196855f747fe2c0856d24b02ca3a5de1b0e633d3998765: Status 404 returned error can't find the container with id f9999745b78b5ae8f0196855f747fe2c0856d24b02ca3a5de1b0e633d3998765 Sep 16 18:47:33.725144 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.725115 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" event={"ID":"322e4d63-e09c-4b27-8681-32b11e13c0cf","Type":"ContainerStarted","Data":"f9999745b78b5ae8f0196855f747fe2c0856d24b02ca3a5de1b0e633d3998765"} Sep 16 18:47:33.726226 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.726205 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" event={"ID":"492c472f-5f87-465e-9534-058f606e5944","Type":"ContainerStarted","Data":"6fc8fa084af3a6ac8bfcd3d5ed1ddd2b4239b703ed1aed1e6dca12cae00c613f"} Sep 16 18:47:33.774979 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.774937 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.775112 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.775038 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:33.775112 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.775082 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 18:47:33.775198 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.775146 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs podName:2a1abe11-738f-42b1-be3c-fbd279e38659 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:34.775129239 +0000 UTC m=+115.098822071 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs") pod "router-default-95c96d4d4-gg962" (UID: "2a1abe11-738f-42b1-be3c-fbd279e38659") : secret "router-metrics-certs-default" not found Sep 16 18:47:33.775198 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.775160 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle podName:2a1abe11-738f-42b1-be3c-fbd279e38659 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:34.775154479 +0000 UTC m=+115.098847306 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle") pod "router-default-95c96d4d4-gg962" (UID: "2a1abe11-738f-42b1-be3c-fbd279e38659") : configmap references non-existent config key: service-ca.crt Sep 16 18:47:33.875574 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:33.875523 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:47:33.875757 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.875683 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 18:47:33.875757 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:33.875753 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert podName:7b5367e2-4fb0-43ec-8046-f1fdf490abeb nodeName:}" failed. No retries permitted until 2026-09-16 18:47:34.87573809 +0000 UTC m=+115.199430920 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqzzv" (UID: "7b5367e2-4fb0-43ec-8046-f1fdf490abeb") : secret "networking-console-plugin-cert" not found Sep 16 18:47:34.683265 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:34.683220 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:34.683757 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:34.683354 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:34.683757 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:34.683434 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls podName:4f1e811d-0e94-42b6-888c-3e2f1952434c nodeName:}" failed. No retries permitted until 2026-09-16 18:47:36.683418563 +0000 UTC m=+117.007111395 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-hms5p" (UID: "4f1e811d-0e94-42b6-888c-3e2f1952434c") : secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:34.783747 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:34.783707 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:34.783950 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:34.783784 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:34.783950 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:34.783896 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 18:47:34.783950 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:34.783926 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle podName:2a1abe11-738f-42b1-be3c-fbd279e38659 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:36.783903305 +0000 UTC m=+117.107596145 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle") pod "router-default-95c96d4d4-gg962" (UID: "2a1abe11-738f-42b1-be3c-fbd279e38659") : configmap references non-existent config key: service-ca.crt Sep 16 18:47:34.784137 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:34.783958 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs podName:2a1abe11-738f-42b1-be3c-fbd279e38659 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:36.783946881 +0000 UTC m=+117.107639710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs") pod "router-default-95c96d4d4-gg962" (UID: "2a1abe11-738f-42b1-be3c-fbd279e38659") : secret "router-metrics-certs-default" not found Sep 16 18:47:34.884913 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:34.884879 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:47:34.885080 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:34.885020 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 18:47:34.885120 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:34.885091 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert podName:7b5367e2-4fb0-43ec-8046-f1fdf490abeb nodeName:}" failed. No retries permitted until 2026-09-16 18:47:36.885073378 +0000 UTC m=+117.208766209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqzzv" (UID: "7b5367e2-4fb0-43ec-8046-f1fdf490abeb") : secret "networking-console-plugin-cert" not found Sep 16 18:47:36.701241 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:36.701188 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:36.701636 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:36.701340 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:36.701636 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:36.701421 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls podName:4f1e811d-0e94-42b6-888c-3e2f1952434c nodeName:}" failed. No retries permitted until 2026-09-16 18:47:40.701405648 +0000 UTC m=+121.025098480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-hms5p" (UID: "4f1e811d-0e94-42b6-888c-3e2f1952434c") : secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:36.736316 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:36.736290 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/0.log" Sep 16 18:47:36.736486 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:36.736329 2563 generic.go:356] "Generic (PLEG): container finished" podID="322e4d63-e09c-4b27-8681-32b11e13c0cf" containerID="dd9db21092722186447f56b2c9b0b724a9eda6fc943e33d24b87735a2b4ede8d" exitCode=255 Sep 16 18:47:36.736486 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:36.736429 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" event={"ID":"322e4d63-e09c-4b27-8681-32b11e13c0cf","Type":"ContainerDied","Data":"dd9db21092722186447f56b2c9b0b724a9eda6fc943e33d24b87735a2b4ede8d"} Sep 16 18:47:36.736704 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:36.736671 2563 scope.go:117] "RemoveContainer" containerID="dd9db21092722186447f56b2c9b0b724a9eda6fc943e33d24b87735a2b4ede8d" Sep 16 18:47:36.738090 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:36.738066 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" event={"ID":"492c472f-5f87-465e-9534-058f606e5944","Type":"ContainerStarted","Data":"5bb070caacd446ac06515faa9fca8a2bcca1c2866ac3a005ba547824e7fa5183"} Sep 16 18:47:36.791828 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:36.791766 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" podStartSLOduration=1.359002075 podStartE2EDuration="3.79174811s" totalImagesPullingTime="2.432746035s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:36.791242186 +0000 UTC m=+117.114935017" watchObservedRunningTime="2026-09-16 18:47:36.79174811 +0000 UTC m=+117.115440958" Sep 16 18:47:36.802752 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:36.802032 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:36.802752 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:36.802149 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 18:47:36.802752 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:36.802213 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs podName:2a1abe11-738f-42b1-be3c-fbd279e38659 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:40.802193576 +0000 UTC m=+121.125886410 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs") pod "router-default-95c96d4d4-gg962" (UID: "2a1abe11-738f-42b1-be3c-fbd279e38659") : secret "router-metrics-certs-default" not found Sep 16 18:47:36.802752 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:36.802352 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:36.802752 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:36.802597 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle podName:2a1abe11-738f-42b1-be3c-fbd279e38659 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:40.802585208 +0000 UTC m=+121.126278040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle") pod "router-default-95c96d4d4-gg962" (UID: "2a1abe11-738f-42b1-be3c-fbd279e38659") : configmap references non-existent config key: service-ca.crt Sep 16 18:47:36.903134 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:36.903098 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:47:36.903272 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:36.903252 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 18:47:36.903329 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:36.903320 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert podName:7b5367e2-4fb0-43ec-8046-f1fdf490abeb nodeName:}" failed. No retries permitted until 2026-09-16 18:47:40.90330475 +0000 UTC m=+121.226997580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqzzv" (UID: "7b5367e2-4fb0-43ec-8046-f1fdf490abeb") : secret "networking-console-plugin-cert" not found Sep 16 18:47:37.216302 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.216266 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw"] Sep 16 18:47:37.218278 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.218260 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw" Sep 16 18:47:37.220810 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.220780 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 16 18:47:37.220979 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.220884 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q84nh\"" Sep 16 18:47:37.221448 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.221412 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 16 18:47:37.247505 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.247473 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw"] Sep 16 18:47:37.306752 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.306714 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mkxv\" (UniqueName: \"kubernetes.io/projected/8c28b886-8c19-4946-9dba-8930bf946190-kube-api-access-7mkxv\") pod \"migrator-8b4d697c5-z9cfw\" (UID: \"8c28b886-8c19-4946-9dba-8930bf946190\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw" Sep 16 18:47:37.407745 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.407705 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mkxv\" (UniqueName: \"kubernetes.io/projected/8c28b886-8c19-4946-9dba-8930bf946190-kube-api-access-7mkxv\") pod \"migrator-8b4d697c5-z9cfw\" (UID: \"8c28b886-8c19-4946-9dba-8930bf946190\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw" Sep 16 18:47:37.420851 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.420792 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mkxv\" (UniqueName: \"kubernetes.io/projected/8c28b886-8c19-4946-9dba-8930bf946190-kube-api-access-7mkxv\") pod \"migrator-8b4d697c5-z9cfw\" (UID: \"8c28b886-8c19-4946-9dba-8930bf946190\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw" Sep 16 18:47:37.456915 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.456878 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wdg6x"] Sep 16 18:47:37.459155 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.459130 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.462782 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.462757 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 18:47:37.462919 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.462765 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 18:47:37.463124 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.463107 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 18:47:37.463207 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.463145 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fk2dc\"" Sep 16 18:47:37.463207 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.463168 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 18:47:37.478606 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.478541 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wdg6x"] Sep 16 18:47:37.508354 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.508321 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.508354 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.508359 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6w4\" (UniqueName: \"kubernetes.io/projected/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-api-access-lx6w4\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.508538 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.508460 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c8b0805-a667-43e8-9a6e-2fef68669628-data-volume\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.508538 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.508503 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c8b0805-a667-43e8-9a6e-2fef68669628-crio-socket\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.508605 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.508566 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.526474 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.526442 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw" Sep 16 18:47:37.609157 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.609125 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c8b0805-a667-43e8-9a6e-2fef68669628-crio-socket\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.609300 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.609207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.609300 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.609248 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1c8b0805-a667-43e8-9a6e-2fef68669628-crio-socket\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.609300 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.609256 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.609539 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.609306 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6w4\" (UniqueName: \"kubernetes.io/projected/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-api-access-lx6w4\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.609539 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:37.609334 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:37.609539 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:37.609392 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm podName:1c8b0805-a667-43e8-9a6e-2fef68669628 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:38.109373482 +0000 UTC m=+118.433066327 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wdg6x" (UID: "1c8b0805-a667-43e8-9a6e-2fef68669628") : configmap "kube-rbac-proxy" not found Sep 16 18:47:37.609539 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:37.609396 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:37.609539 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:37.609471 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls podName:1c8b0805-a667-43e8-9a6e-2fef68669628 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:38.10944897 +0000 UTC m=+118.433141823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wdg6x" (UID: "1c8b0805-a667-43e8-9a6e-2fef68669628") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:37.609539 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.609406 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c8b0805-a667-43e8-9a6e-2fef68669628-data-volume\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.609745 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.609659 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1c8b0805-a667-43e8-9a6e-2fef68669628-data-volume\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.627368 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.627343 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6w4\" (UniqueName: \"kubernetes.io/projected/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-api-access-lx6w4\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:37.743142 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.743074 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/1.log" Sep 16 18:47:37.743518 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.743501 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/0.log" Sep 16 18:47:37.743559 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.743538 2563 generic.go:356] "Generic (PLEG): container finished" podID="322e4d63-e09c-4b27-8681-32b11e13c0cf" containerID="d865819b6186ee1a453cc2d54b997f3eca186db3ef858cfbbf69fe4542aa2a67" exitCode=255 Sep 16 18:47:37.743622 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.743609 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" event={"ID":"322e4d63-e09c-4b27-8681-32b11e13c0cf","Type":"ContainerDied","Data":"d865819b6186ee1a453cc2d54b997f3eca186db3ef858cfbbf69fe4542aa2a67"} Sep 16 18:47:37.743660 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.743650 2563 scope.go:117] "RemoveContainer" containerID="dd9db21092722186447f56b2c9b0b724a9eda6fc943e33d24b87735a2b4ede8d" Sep 16 18:47:37.743911 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.743887 2563 scope.go:117] "RemoveContainer" containerID="d865819b6186ee1a453cc2d54b997f3eca186db3ef858cfbbf69fe4542aa2a67" Sep 16 18:47:37.744099 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:37.744077 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-f2hw9_openshift-console-operator(322e4d63-e09c-4b27-8681-32b11e13c0cf)\"" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" podUID="322e4d63-e09c-4b27-8681-32b11e13c0cf" Sep 16 18:47:37.804566 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:37.803038 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw"] Sep 16 18:47:37.808269 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:47:37.808237 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c28b886_8c19_4946_9dba_8930bf946190.slice/crio-ff1f3ff2fb542a681c618fbd4b60a4c562a7c467e1161790ae65f480f307c799 WatchSource:0}: Error finding container ff1f3ff2fb542a681c618fbd4b60a4c562a7c467e1161790ae65f480f307c799: Status 404 returned error can't find the container with id ff1f3ff2fb542a681c618fbd4b60a4c562a7c467e1161790ae65f480f307c799 Sep 16 18:47:38.113461 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:38.113421 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:38.113641 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:38.113469 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:38.113641 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:38.113564 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:38.113641 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:38.113569 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:38.113641 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:38.113625 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm podName:1c8b0805-a667-43e8-9a6e-2fef68669628 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:39.113610622 +0000 UTC m=+119.437303451 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wdg6x" (UID: "1c8b0805-a667-43e8-9a6e-2fef68669628") : configmap "kube-rbac-proxy" not found Sep 16 18:47:38.113641 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:38.113640 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls podName:1c8b0805-a667-43e8-9a6e-2fef68669628 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:39.113634187 +0000 UTC m=+119.437327013 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wdg6x" (UID: "1c8b0805-a667-43e8-9a6e-2fef68669628") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:38.747667 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:38.747633 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/1.log" Sep 16 18:47:38.748150 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:38.748074 2563 scope.go:117] "RemoveContainer" containerID="d865819b6186ee1a453cc2d54b997f3eca186db3ef858cfbbf69fe4542aa2a67" Sep 16 18:47:38.748269 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:38.748242 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-f2hw9_openshift-console-operator(322e4d63-e09c-4b27-8681-32b11e13c0cf)\"" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" podUID="322e4d63-e09c-4b27-8681-32b11e13c0cf" Sep 16 18:47:38.749186 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:38.749156 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw" event={"ID":"8c28b886-8c19-4946-9dba-8930bf946190","Type":"ContainerStarted","Data":"ff1f3ff2fb542a681c618fbd4b60a4c562a7c467e1161790ae65f480f307c799"} Sep 16 18:47:39.122772 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.122729 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:39.122979 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:39.122872 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:39.122979 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.122891 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:39.122979 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:39.122927 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls podName:1c8b0805-a667-43e8-9a6e-2fef68669628 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:41.122912738 +0000 UTC m=+121.446605582 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wdg6x" (UID: "1c8b0805-a667-43e8-9a6e-2fef68669628") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:39.122979 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:39.122952 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:39.123136 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:39.123002 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm podName:1c8b0805-a667-43e8-9a6e-2fef68669628 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:41.12298994 +0000 UTC m=+121.446682790 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wdg6x" (UID: "1c8b0805-a667-43e8-9a6e-2fef68669628") : configmap "kube-rbac-proxy" not found Sep 16 18:47:39.469790 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.469711 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xz5fl"] Sep 16 18:47:39.471720 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.471705 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" Sep 16 18:47:39.476597 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.476561 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 16 18:47:39.476735 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.476643 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 16 18:47:39.477088 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.477066 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 16 18:47:39.477223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.477135 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cv7zw\"" Sep 16 18:47:39.477223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.477141 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 16 18:47:39.488526 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.488496 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xz5fl"] Sep 16 18:47:39.526059 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.526023 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1f8fb7a0-0e40-4411-873f-089a69695195-signing-key\") pod \"service-ca-568869fdc5-xz5fl\" (UID: \"1f8fb7a0-0e40-4411-873f-089a69695195\") " pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" Sep 16 18:47:39.526059 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.526060 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgq72\" (UniqueName: \"kubernetes.io/projected/1f8fb7a0-0e40-4411-873f-089a69695195-kube-api-access-kgq72\") pod \"service-ca-568869fdc5-xz5fl\" (UID: \"1f8fb7a0-0e40-4411-873f-089a69695195\") " pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" Sep 16 18:47:39.526277 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.526128 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f8fb7a0-0e40-4411-873f-089a69695195-signing-cabundle\") pod \"service-ca-568869fdc5-xz5fl\" (UID: \"1f8fb7a0-0e40-4411-873f-089a69695195\") " pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" Sep 16 18:47:39.627277 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.627241 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f8fb7a0-0e40-4411-873f-089a69695195-signing-cabundle\") pod \"service-ca-568869fdc5-xz5fl\" (UID: \"1f8fb7a0-0e40-4411-873f-089a69695195\") " pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" Sep 16 18:47:39.627451 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.627390 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1f8fb7a0-0e40-4411-873f-089a69695195-signing-key\") pod \"service-ca-568869fdc5-xz5fl\" (UID: \"1f8fb7a0-0e40-4411-873f-089a69695195\") " pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" Sep 16 18:47:39.627451 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.627419 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgq72\" (UniqueName: \"kubernetes.io/projected/1f8fb7a0-0e40-4411-873f-089a69695195-kube-api-access-kgq72\") pod \"service-ca-568869fdc5-xz5fl\" (UID: \"1f8fb7a0-0e40-4411-873f-089a69695195\") " pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" Sep 16 18:47:39.628088 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.628063 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1f8fb7a0-0e40-4411-873f-089a69695195-signing-cabundle\") pod \"service-ca-568869fdc5-xz5fl\" (UID: \"1f8fb7a0-0e40-4411-873f-089a69695195\") " pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" Sep 16 18:47:39.629920 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.629895 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1f8fb7a0-0e40-4411-873f-089a69695195-signing-key\") pod \"service-ca-568869fdc5-xz5fl\" (UID: \"1f8fb7a0-0e40-4411-873f-089a69695195\") " pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" Sep 16 18:47:39.640828 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.640801 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgq72\" (UniqueName: \"kubernetes.io/projected/1f8fb7a0-0e40-4411-873f-089a69695195-kube-api-access-kgq72\") pod \"service-ca-568869fdc5-xz5fl\" (UID: \"1f8fb7a0-0e40-4411-873f-089a69695195\") " pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" Sep 16 18:47:39.754224 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.754194 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw" event={"ID":"8c28b886-8c19-4946-9dba-8930bf946190","Type":"ContainerStarted","Data":"2b27409f2c8ea27c06cecbca02fa98276fdcd82cdb61a73901a4ddfa7d3370b7"} Sep 16 18:47:39.754593 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.754249 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw" event={"ID":"8c28b886-8c19-4946-9dba-8930bf946190","Type":"ContainerStarted","Data":"a43c9b0a9bd510f0f69dd302a4b27da9be4c80c11c9d335def42041edd3500ab"} Sep 16 18:47:39.780311 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:39.780269 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" Sep 16 18:47:40.072182 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:40.072087 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z9cfw" podStartSLOduration=2.056833695 podStartE2EDuration="3.072068131s" totalImagesPullingTime="1.015234436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:39.785506305 +0000 UTC m=+120.109199153" watchObservedRunningTime="2026-09-16 18:47:40.072068131 +0000 UTC m=+120.395760986" Sep 16 18:47:40.073007 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:40.072934 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-xz5fl"] Sep 16 18:47:40.076549 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:47:40.076524 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f8fb7a0_0e40_4411_873f_089a69695195.slice/crio-be934e290d7beda1d1e15929edb802adda896198160c5c685d37896b8635dd48 WatchSource:0}: Error finding container be934e290d7beda1d1e15929edb802adda896198160c5c685d37896b8635dd48: Status 404 returned error can't find the container with id be934e290d7beda1d1e15929edb802adda896198160c5c685d37896b8635dd48 Sep 16 18:47:40.143415 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:40.143386 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lh5wh_f2231582-bd0c-440b-9841-bd4740794c8c/dns-node-resolver/0.log" Sep 16 18:47:40.736884 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:40.736841 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:40.737075 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:40.737000 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:40.737124 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:40.737085 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls podName:4f1e811d-0e94-42b6-888c-3e2f1952434c nodeName:}" failed. No retries permitted until 2026-09-16 18:47:48.737065895 +0000 UTC m=+129.060758726 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-hms5p" (UID: "4f1e811d-0e94-42b6-888c-3e2f1952434c") : secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:40.758381 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:40.758335 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" event={"ID":"1f8fb7a0-0e40-4411-873f-089a69695195","Type":"ContainerStarted","Data":"be934e290d7beda1d1e15929edb802adda896198160c5c685d37896b8635dd48"} Sep 16 18:47:40.837498 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:40.837445 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:40.837676 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:40.837611 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 16 18:47:40.837676 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:40.837648 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:40.837779 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:40.837683 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs podName:2a1abe11-738f-42b1-be3c-fbd279e38659 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:48.837663442 +0000 UTC m=+129.161356274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs") pod "router-default-95c96d4d4-gg962" (UID: "2a1abe11-738f-42b1-be3c-fbd279e38659") : secret "router-metrics-certs-default" not found Sep 16 18:47:40.837779 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:40.837732 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle podName:2a1abe11-738f-42b1-be3c-fbd279e38659 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:48.837721433 +0000 UTC m=+129.161414274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle") pod "router-default-95c96d4d4-gg962" (UID: "2a1abe11-738f-42b1-be3c-fbd279e38659") : configmap references non-existent config key: service-ca.crt Sep 16 18:47:40.938454 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:40.938396 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:47:40.938642 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:40.938571 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 18:47:40.938705 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:40.938657 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert podName:7b5367e2-4fb0-43ec-8046-f1fdf490abeb nodeName:}" failed. No retries permitted until 2026-09-16 18:47:48.93863498 +0000 UTC m=+129.262327823 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqzzv" (UID: "7b5367e2-4fb0-43ec-8046-f1fdf490abeb") : secret "networking-console-plugin-cert" not found Sep 16 18:47:40.944793 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:40.944761 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h946g_01d77fab-c94e-46ea-b0e8-1b2b02f5d977/node-ca/0.log" Sep 16 18:47:41.140463 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:41.140428 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:41.140639 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:41.140481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:41.140639 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:41.140551 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:41.140639 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:41.140594 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:41.140639 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:41.140604 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm podName:1c8b0805-a667-43e8-9a6e-2fef68669628 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:45.140589115 +0000 UTC m=+125.464281942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wdg6x" (UID: "1c8b0805-a667-43e8-9a6e-2fef68669628") : configmap "kube-rbac-proxy" not found Sep 16 18:47:41.140794 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:41.140666 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls podName:1c8b0805-a667-43e8-9a6e-2fef68669628 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:45.140652536 +0000 UTC m=+125.464345378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wdg6x" (UID: "1c8b0805-a667-43e8-9a6e-2fef68669628") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:41.737932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:41.737905 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8vrw2_d1bc280d-ee46-48fe-a939-7b6d82bb1c92/serve-healthcheck-canary/0.log" Sep 16 18:47:41.762749 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:41.762715 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" event={"ID":"1f8fb7a0-0e40-4411-873f-089a69695195","Type":"ContainerStarted","Data":"7bce925999773bb032178817f43a9dd37e4e545436e1b308723136118a429892"} Sep 16 18:47:41.795243 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:41.795155 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-xz5fl" podStartSLOduration=1.255155168 podStartE2EDuration="2.79513625s" totalImagesPullingTime="1.539981082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:41.794531933 +0000 UTC m=+122.118224783" watchObservedRunningTime="2026-09-16 18:47:41.79513625 +0000 UTC m=+122.118829099" Sep 16 18:47:42.334150 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:42.334102 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-z9cfw_8c28b886-8c19-4946-9dba-8930bf946190/migrator/0.log" Sep 16 18:47:42.534376 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:42.534338 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-z9cfw_8c28b886-8c19-4946-9dba-8930bf946190/graceful-termination/0.log" Sep 16 18:47:42.736940 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:42.736826 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-4rpwl_492c472f-5f87-465e-9534-058f606e5944/kube-storage-version-migrator-operator/0.log" Sep 16 18:47:43.397222 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:43.397175 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:43.397692 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:43.397253 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:47:43.397766 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:43.397702 2563 scope.go:117] "RemoveContainer" containerID="d865819b6186ee1a453cc2d54b997f3eca186db3ef858cfbbf69fe4542aa2a67" Sep 16 18:47:43.397936 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:43.397909 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-f2hw9_openshift-console-operator(322e4d63-e09c-4b27-8681-32b11e13c0cf)\"" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" podUID="322e4d63-e09c-4b27-8681-32b11e13c0cf" Sep 16 18:47:45.178665 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:45.178616 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:45.179133 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:45.178679 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:45.179133 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:45.178756 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 18:47:45.179133 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:45.178767 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 18:47:45.179133 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:45.178865 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm podName:1c8b0805-a667-43e8-9a6e-2fef68669628 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:53.178807346 +0000 UTC m=+133.502500177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wdg6x" (UID: "1c8b0805-a667-43e8-9a6e-2fef68669628") : configmap "kube-rbac-proxy" not found Sep 16 18:47:45.179133 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:45.178892 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls podName:1c8b0805-a667-43e8-9a6e-2fef68669628 nodeName:}" failed. No retries permitted until 2026-09-16 18:47:53.178879303 +0000 UTC m=+133.502572153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wdg6x" (UID: "1c8b0805-a667-43e8-9a6e-2fef68669628") : secret "insights-runtime-extractor-tls" not found Sep 16 18:47:48.813367 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:48.813318 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:47:48.813930 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:48.813491 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:48.813930 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:48.813585 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls podName:4f1e811d-0e94-42b6-888c-3e2f1952434c nodeName:}" failed. No retries permitted until 2026-09-16 18:48:04.8135619 +0000 UTC m=+145.137254732 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-hms5p" (UID: "4f1e811d-0e94-42b6-888c-3e2f1952434c") : secret "cluster-monitoring-operator-tls" not found Sep 16 18:47:48.913962 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:48.913918 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:48.913962 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:48.913965 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:48.914167 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:48.914112 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle podName:2a1abe11-738f-42b1-be3c-fbd279e38659 nodeName:}" failed. No retries permitted until 2026-09-16 18:48:04.914092561 +0000 UTC m=+145.237785391 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle") pod "router-default-95c96d4d4-gg962" (UID: "2a1abe11-738f-42b1-be3c-fbd279e38659") : configmap references non-existent config key: service-ca.crt Sep 16 18:47:48.916429 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:48.916406 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a1abe11-738f-42b1-be3c-fbd279e38659-metrics-certs\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:47:49.014763 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:49.014719 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:47:49.014950 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:49.014907 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 16 18:47:49.015024 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:49.015012 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert podName:7b5367e2-4fb0-43ec-8046-f1fdf490abeb nodeName:}" failed. No retries permitted until 2026-09-16 18:48:05.014992039 +0000 UTC m=+145.338684867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-nqzzv" (UID: "7b5367e2-4fb0-43ec-8046-f1fdf490abeb") : secret "networking-console-plugin-cert" not found Sep 16 18:47:49.922009 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:49.921962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:47:49.922417 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:49.922138 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 16 18:47:49.922417 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:49.922224 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs podName:650deff9-b845-4ef3-acad-6bdee9667025 nodeName:}" failed. No retries permitted until 2026-09-16 18:49:51.922202545 +0000 UTC m=+252.245895386 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs") pod "network-metrics-daemon-lq68v" (UID: "650deff9-b845-4ef3-acad-6bdee9667025") : secret "metrics-daemon-secret" not found Sep 16 18:47:53.250196 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:53.250153 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:53.250569 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:53.250204 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:53.250803 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:53.250784 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1c8b0805-a667-43e8-9a6e-2fef68669628-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:53.252577 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:53.252552 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1c8b0805-a667-43e8-9a6e-2fef68669628-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wdg6x\" (UID: \"1c8b0805-a667-43e8-9a6e-2fef68669628\") " pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:53.371775 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:53.371734 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fk2dc\"" Sep 16 18:47:53.378903 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:53.378870 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wdg6x" Sep 16 18:47:53.659950 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:47:53.659916 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8b0805_a667_43e8_9a6e_2fef68669628.slice/crio-0127f79466014be110edcfcb31dda25c7a90a6e49db6da938edf8e0a97986089 WatchSource:0}: Error finding container 0127f79466014be110edcfcb31dda25c7a90a6e49db6da938edf8e0a97986089: Status 404 returned error can't find the container with id 0127f79466014be110edcfcb31dda25c7a90a6e49db6da938edf8e0a97986089 Sep 16 18:47:53.663747 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:53.663722 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wdg6x"] Sep 16 18:47:53.799931 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:53.799893 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wdg6x" event={"ID":"1c8b0805-a667-43e8-9a6e-2fef68669628","Type":"ContainerStarted","Data":"c6e036120488279a527f74edca66ec3b7cd92eb393e7fc70d6e88e9a6db44e7e"} Sep 16 18:47:53.800103 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:53.799955 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wdg6x" event={"ID":"1c8b0805-a667-43e8-9a6e-2fef68669628","Type":"ContainerStarted","Data":"0127f79466014be110edcfcb31dda25c7a90a6e49db6da938edf8e0a97986089"} Sep 16 18:47:54.804938 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:54.804897 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wdg6x" event={"ID":"1c8b0805-a667-43e8-9a6e-2fef68669628","Type":"ContainerStarted","Data":"35a68f7cd987c73e3ad1f485a1aa9ebf55d91228fde9f6c71785d73efc0d4f86"} Sep 16 18:47:55.809962 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:55.809926 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wdg6x" event={"ID":"1c8b0805-a667-43e8-9a6e-2fef68669628","Type":"ContainerStarted","Data":"79bcfa452537918906c21b106ebabde607b3fdb40faae78a58e25cb678296591"} Sep 16 18:47:55.833661 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:55.833603 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wdg6x" podStartSLOduration=17.138559877 podStartE2EDuration="18.833583922s" totalImagesPullingTime="1.695024045s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:47:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:55.832762229 +0000 UTC m=+136.156455090" watchObservedRunningTime="2026-09-16 18:47:55.833583922 +0000 UTC m=+136.157276771" Sep 16 18:47:56.331724 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:56.331692 2563 scope.go:117] "RemoveContainer" containerID="d865819b6186ee1a453cc2d54b997f3eca186db3ef858cfbbf69fe4542aa2a67" Sep 16 18:47:56.815006 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:56.814976 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 18:47:56.815500 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:56.815481 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/1.log" Sep 16 18:47:56.815574 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:56.815525 2563 generic.go:356] "Generic (PLEG): container finished" podID="322e4d63-e09c-4b27-8681-32b11e13c0cf" containerID="d1b4a70d71d40d0d60d1c41f454fae1f8250adbd3029c79e2d6cdeb510c44a59" exitCode=255 Sep 16 18:47:56.815629 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:56.815603 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" event={"ID":"322e4d63-e09c-4b27-8681-32b11e13c0cf","Type":"ContainerDied","Data":"d1b4a70d71d40d0d60d1c41f454fae1f8250adbd3029c79e2d6cdeb510c44a59"} Sep 16 18:47:56.815685 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:56.815664 2563 scope.go:117] "RemoveContainer" containerID="d865819b6186ee1a453cc2d54b997f3eca186db3ef858cfbbf69fe4542aa2a67" Sep 16 18:47:56.816068 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:56.816048 2563 scope.go:117] "RemoveContainer" containerID="d1b4a70d71d40d0d60d1c41f454fae1f8250adbd3029c79e2d6cdeb510c44a59" Sep 16 18:47:56.816257 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:47:56.816238 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-f2hw9_openshift-console-operator(322e4d63-e09c-4b27-8681-32b11e13c0cf)\"" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" podUID="322e4d63-e09c-4b27-8681-32b11e13c0cf" Sep 16 18:47:57.820935 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:47:57.820905 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 18:48:03.397033 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:03.396985 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:48:03.397460 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:03.397059 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:48:03.397460 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:03.397416 2563 scope.go:117] "RemoveContainer" containerID="d1b4a70d71d40d0d60d1c41f454fae1f8250adbd3029c79e2d6cdeb510c44a59" Sep 16 18:48:03.397607 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:48:03.397586 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-f2hw9_openshift-console-operator(322e4d63-e09c-4b27-8681-32b11e13c0cf)\"" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" podUID="322e4d63-e09c-4b27-8681-32b11e13c0cf" Sep 16 18:48:04.844207 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:04.844163 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:48:04.846934 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:04.846902 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/4f1e811d-0e94-42b6-888c-3e2f1952434c-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-hms5p\" (UID: \"4f1e811d-0e94-42b6-888c-3e2f1952434c\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:48:04.944971 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:04.944922 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:48:04.945575 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:04.945551 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a1abe11-738f-42b1-be3c-fbd279e38659-service-ca-bundle\") pod \"router-default-95c96d4d4-gg962\" (UID: \"2a1abe11-738f-42b1-be3c-fbd279e38659\") " pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:48:05.045942 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.045901 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:48:05.048271 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.048245 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b5367e2-4fb0-43ec-8046-f1fdf490abeb-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-nqzzv\" (UID: \"7b5367e2-4fb0-43ec-8046-f1fdf490abeb\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:48:05.080709 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.080681 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qm9dd\"" Sep 16 18:48:05.088517 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.088497 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" Sep 16 18:48:05.192931 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.192901 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lr9kh\"" Sep 16 18:48:05.201284 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.201258 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:48:05.301001 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.300967 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-z6hvf\"" Sep 16 18:48:05.308924 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.308903 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" Sep 16 18:48:05.359323 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.359268 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p"] Sep 16 18:48:05.365816 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:48:05.365784 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1e811d_0e94_42b6_888c_3e2f1952434c.slice/crio-8e6029dc27166a584a29030f4bad560550494a26bb3ac2399bf2598b07f9ac5b WatchSource:0}: Error finding container 8e6029dc27166a584a29030f4bad560550494a26bb3ac2399bf2598b07f9ac5b: Status 404 returned error can't find the container with id 8e6029dc27166a584a29030f4bad560550494a26bb3ac2399bf2598b07f9ac5b Sep 16 18:48:05.472996 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.472964 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-95c96d4d4-gg962"] Sep 16 18:48:05.475964 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:48:05.475935 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a1abe11_738f_42b1_be3c_fbd279e38659.slice/crio-9e86034f7ff054d059680b978199a7ae34b57a49f246512bd9b5e71fe452a60f WatchSource:0}: Error finding container 9e86034f7ff054d059680b978199a7ae34b57a49f246512bd9b5e71fe452a60f: Status 404 returned error can't find the container with id 9e86034f7ff054d059680b978199a7ae34b57a49f246512bd9b5e71fe452a60f Sep 16 18:48:05.583993 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.583954 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv"] Sep 16 18:48:05.588481 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:48:05.588451 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5367e2_4fb0_43ec_8046_f1fdf490abeb.slice/crio-575ec0d6608c59ec4350efe5b1c7b2b538e4d26bc253885be2919f84f35fb40e WatchSource:0}: Error finding container 575ec0d6608c59ec4350efe5b1c7b2b538e4d26bc253885be2919f84f35fb40e: Status 404 returned error can't find the container with id 575ec0d6608c59ec4350efe5b1c7b2b538e4d26bc253885be2919f84f35fb40e Sep 16 18:48:05.851760 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.851720 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" event={"ID":"7b5367e2-4fb0-43ec-8046-f1fdf490abeb","Type":"ContainerStarted","Data":"575ec0d6608c59ec4350efe5b1c7b2b538e4d26bc253885be2919f84f35fb40e"} Sep 16 18:48:05.853569 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.853528 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-95c96d4d4-gg962" event={"ID":"2a1abe11-738f-42b1-be3c-fbd279e38659","Type":"ContainerStarted","Data":"e33b35da6b96e9064076a8a180ad999c716ef2a4acf768267c3b8e51fd42deb4"} Sep 16 18:48:05.853709 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.853584 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-95c96d4d4-gg962" event={"ID":"2a1abe11-738f-42b1-be3c-fbd279e38659","Type":"ContainerStarted","Data":"9e86034f7ff054d059680b978199a7ae34b57a49f246512bd9b5e71fe452a60f"} Sep 16 18:48:05.855104 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.855072 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" event={"ID":"4f1e811d-0e94-42b6-888c-3e2f1952434c","Type":"ContainerStarted","Data":"8e6029dc27166a584a29030f4bad560550494a26bb3ac2399bf2598b07f9ac5b"} Sep 16 18:48:05.878780 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:05.878719 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-95c96d4d4-gg962" podStartSLOduration=32.878702421 podStartE2EDuration="32.878702421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:05.877819057 +0000 UTC m=+146.201511908" watchObservedRunningTime="2026-09-16 18:48:05.878702421 +0000 UTC m=+146.202395260" Sep 16 18:48:06.201704 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:06.201610 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:48:06.204972 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:06.204740 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:48:06.859403 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:06.859365 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:48:06.860824 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:06.860802 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-95c96d4d4-gg962" Sep 16 18:48:07.864201 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:07.864165 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" event={"ID":"4f1e811d-0e94-42b6-888c-3e2f1952434c","Type":"ContainerStarted","Data":"30309d2e0863a4704bed9cb43b3a9988794ca9a92a6163398ef46c6640e30274"} Sep 16 18:48:07.865903 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:07.865873 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" event={"ID":"7b5367e2-4fb0-43ec-8046-f1fdf490abeb","Type":"ContainerStarted","Data":"251272531b04bf574e94658bb0009bdf6ce409c451f187a30098e879347ea3ab"} Sep 16 18:48:07.917519 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:07.917463 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-hms5p" podStartSLOduration=34.044232109 podStartE2EDuration="35.917442826s" totalImagesPullingTime="1.873210717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:07.90871536 +0000 UTC m=+148.232408208" watchObservedRunningTime="2026-09-16 18:48:07.917442826 +0000 UTC m=+148.241135717" Sep 16 18:48:07.950648 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:07.950592 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-nqzzv" podStartSLOduration=33.29789409 podStartE2EDuration="34.950567997s" totalImagesPullingTime="1.652673907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:07.948718932 +0000 UTC m=+148.272411786" watchObservedRunningTime="2026-09-16 18:48:07.950567997 +0000 UTC m=+148.274260847" Sep 16 18:48:09.995098 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:09.995062 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xgwks"] Sep 16 18:48:10.023560 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.023522 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xgwks"] Sep 16 18:48:10.023730 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.023643 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.027102 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.027077 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 16 18:48:10.027426 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.027410 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4lpjf\"" Sep 16 18:48:10.027672 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.027657 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 18:48:10.027727 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.027670 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 16 18:48:10.085230 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.085188 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a47d1919-84c7-4649-af42-ed5e2c37c05a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.085230 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.085233 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a47d1919-84c7-4649-af42-ed5e2c37c05a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.085467 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.085295 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a47d1919-84c7-4649-af42-ed5e2c37c05a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.085467 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.085344 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d9tj\" (UniqueName: \"kubernetes.io/projected/a47d1919-84c7-4649-af42-ed5e2c37c05a-kube-api-access-7d9tj\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.185952 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.185907 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a47d1919-84c7-4649-af42-ed5e2c37c05a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.185952 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.185955 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a47d1919-84c7-4649-af42-ed5e2c37c05a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.186214 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.185989 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a47d1919-84c7-4649-af42-ed5e2c37c05a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.186214 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.186010 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d9tj\" (UniqueName: \"kubernetes.io/projected/a47d1919-84c7-4649-af42-ed5e2c37c05a-kube-api-access-7d9tj\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.186214 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:48:10.186058 2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Sep 16 18:48:10.186214 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:48:10.186140 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a47d1919-84c7-4649-af42-ed5e2c37c05a-prometheus-operator-tls podName:a47d1919-84c7-4649-af42-ed5e2c37c05a nodeName:}" failed. No retries permitted until 2026-09-16 18:48:10.686117999 +0000 UTC m=+151.009810849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a47d1919-84c7-4649-af42-ed5e2c37c05a-prometheus-operator-tls") pod "prometheus-operator-68bcd7f987-xgwks" (UID: "a47d1919-84c7-4649-af42-ed5e2c37c05a") : secret "prometheus-operator-tls" not found Sep 16 18:48:10.186757 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.186728 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a47d1919-84c7-4649-af42-ed5e2c37c05a-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.188598 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.188576 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a47d1919-84c7-4649-af42-ed5e2c37c05a-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.196232 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.196200 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d9tj\" (UniqueName: \"kubernetes.io/projected/a47d1919-84c7-4649-af42-ed5e2c37c05a-kube-api-access-7d9tj\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.688587 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.688535 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a47d1919-84c7-4649-af42-ed5e2c37c05a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.690986 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.690965 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a47d1919-84c7-4649-af42-ed5e2c37c05a-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-xgwks\" (UID: \"a47d1919-84c7-4649-af42-ed5e2c37c05a\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:10.933637 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:10.933595 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" Sep 16 18:48:11.214693 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:11.214611 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-xgwks"] Sep 16 18:48:11.217871 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:48:11.217844 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda47d1919_84c7_4649_af42_ed5e2c37c05a.slice/crio-59424a3026bbbdf06e8ad7cfebb6ca4aa0e0b198626127bf0cd10ba02bdf49bc WatchSource:0}: Error finding container 59424a3026bbbdf06e8ad7cfebb6ca4aa0e0b198626127bf0cd10ba02bdf49bc: Status 404 returned error can't find the container with id 59424a3026bbbdf06e8ad7cfebb6ca4aa0e0b198626127bf0cd10ba02bdf49bc Sep 16 18:48:11.879954 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:11.879913 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" event={"ID":"a47d1919-84c7-4649-af42-ed5e2c37c05a","Type":"ContainerStarted","Data":"59424a3026bbbdf06e8ad7cfebb6ca4aa0e0b198626127bf0cd10ba02bdf49bc"} Sep 16 18:48:12.885101 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:12.885056 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" event={"ID":"a47d1919-84c7-4649-af42-ed5e2c37c05a","Type":"ContainerStarted","Data":"92b15e87b11020e574bf1dd6b2900d59bbb14911c5c48a76ed4ff9af98a724d8"} Sep 16 18:48:12.885493 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:12.885127 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" event={"ID":"a47d1919-84c7-4649-af42-ed5e2c37c05a","Type":"ContainerStarted","Data":"36f755dccc7af6452afb16137a43d4d27b97dc4b37bd4237aa0de2bc063b4b24"} Sep 16 18:48:12.905167 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:12.905105 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-xgwks" podStartSLOduration=2.732622422 podStartE2EDuration="3.905087991s" totalImagesPullingTime="1.172465569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:12.904172917 +0000 UTC m=+153.227865766" watchObservedRunningTime="2026-09-16 18:48:12.905087991 +0000 UTC m=+153.228780841" Sep 16 18:48:14.331487 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.331459 2563 scope.go:117] "RemoveContainer" containerID="d1b4a70d71d40d0d60d1c41f454fae1f8250adbd3029c79e2d6cdeb510c44a59" Sep 16 18:48:14.331962 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:48:14.331626 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-f2hw9_openshift-console-operator(322e4d63-e09c-4b27-8681-32b11e13c0cf)\"" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" podUID="322e4d63-e09c-4b27-8681-32b11e13c0cf" Sep 16 18:48:14.434723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.434692 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb"] Sep 16 18:48:14.436923 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.436904 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.439132 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.439108 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 16 18:48:14.439636 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.439618 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 16 18:48:14.439715 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.439642 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pz7fb\"" Sep 16 18:48:14.439925 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.439907 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 16 18:48:14.451531 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.451509 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb"] Sep 16 18:48:14.457320 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.457297 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pdkl8"] Sep 16 18:48:14.459635 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.459619 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.462123 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.462103 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 18:48:14.462247 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.462149 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 18:48:14.462247 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.462105 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 18:48:14.462718 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.462701 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rtrnx\"" Sep 16 18:48:14.521412 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521374 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.521616 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.521616 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521501 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70580a6d-7e35-4eb0-85eb-65239da83d0b-root\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.521616 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521526 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-tls\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.521616 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521552 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.521823 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521625 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xjhj\" (UniqueName: \"kubernetes.io/projected/70580a6d-7e35-4eb0-85eb-65239da83d0b-kube-api-access-6xjhj\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.521823 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521673 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40efaf55-090f-49b5-97fb-c3a3fef1e69f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.521823 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521697 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70580a6d-7e35-4eb0-85eb-65239da83d0b-metrics-client-ca\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.521823 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521724 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmbwk\" (UniqueName: \"kubernetes.io/projected/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-api-access-pmbwk\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.521823 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521762 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-textfile\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.521823 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521794 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-wtmp\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.521823 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521819 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/40efaf55-090f-49b5-97fb-c3a3fef1e69f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.522173 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521859 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.522173 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521886 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.522173 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.521923 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70580a6d-7e35-4eb0-85eb-65239da83d0b-sys\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.622522 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622452 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmbwk\" (UniqueName: \"kubernetes.io/projected/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-api-access-pmbwk\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.622522 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622488 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-textfile\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.622522 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622507 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-wtmp\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.622795 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622527 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/40efaf55-090f-49b5-97fb-c3a3fef1e69f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.622795 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622673 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.622795 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622686 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-wtmp\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.622795 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622723 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.622795 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622761 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70580a6d-7e35-4eb0-85eb-65239da83d0b-sys\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.623070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622802 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.623070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622890 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.623070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622906 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-textfile\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.623070 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:48:14.622916 2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Sep 16 18:48:14.623070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622953 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/40efaf55-090f-49b5-97fb-c3a3fef1e69f-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.623070 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:48:14.622976 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-tls podName:40efaf55-090f-49b5-97fb-c3a3fef1e69f nodeName:}" failed. No retries permitted until 2026-09-16 18:48:15.122956018 +0000 UTC m=+155.446648848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-k6mjb" (UID: "40efaf55-090f-49b5-97fb-c3a3fef1e69f") : secret "kube-state-metrics-tls" not found Sep 16 18:48:14.623070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622960 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70580a6d-7e35-4eb0-85eb-65239da83d0b-root\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.623070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.622916 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/70580a6d-7e35-4eb0-85eb-65239da83d0b-root\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.623070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.623021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-tls\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.623070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.623048 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.623585 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.623102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xjhj\" (UniqueName: \"kubernetes.io/projected/70580a6d-7e35-4eb0-85eb-65239da83d0b-kube-api-access-6xjhj\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.623585 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.623145 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40efaf55-090f-49b5-97fb-c3a3fef1e69f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.623585 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.623172 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70580a6d-7e35-4eb0-85eb-65239da83d0b-metrics-client-ca\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.623585 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.623189 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/70580a6d-7e35-4eb0-85eb-65239da83d0b-sys\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.623585 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:48:14.623386 2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 18:48:14.623585 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.623430 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-accelerators-collector-config\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.623585 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:48:14.623452 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-tls podName:70580a6d-7e35-4eb0-85eb-65239da83d0b nodeName:}" failed. No retries permitted until 2026-09-16 18:48:15.123437126 +0000 UTC m=+155.447129966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-tls") pod "node-exporter-pdkl8" (UID: "70580a6d-7e35-4eb0-85eb-65239da83d0b") : secret "node-exporter-tls" not found Sep 16 18:48:14.623585 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.623558 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.624035 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.623921 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70580a6d-7e35-4eb0-85eb-65239da83d0b-metrics-client-ca\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.624121 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.624040 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40efaf55-090f-49b5-97fb-c3a3fef1e69f-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.625473 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.625453 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.625779 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.625758 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:14.635206 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.635182 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xjhj\" (UniqueName: \"kubernetes.io/projected/70580a6d-7e35-4eb0-85eb-65239da83d0b-kube-api-access-6xjhj\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:14.635331 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:14.635300 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmbwk\" (UniqueName: \"kubernetes.io/projected/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-api-access-pmbwk\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:15.126525 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:15.126487 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:15.126724 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:15.126584 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-tls\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:15.128891 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:15.128865 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/70580a6d-7e35-4eb0-85eb-65239da83d0b-node-exporter-tls\") pod \"node-exporter-pdkl8\" (UID: \"70580a6d-7e35-4eb0-85eb-65239da83d0b\") " pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:15.129027 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:15.128946 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40efaf55-090f-49b5-97fb-c3a3fef1e69f-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-k6mjb\" (UID: \"40efaf55-090f-49b5-97fb-c3a3fef1e69f\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:15.346707 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:15.346658 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" Sep 16 18:48:15.368810 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:15.368777 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pdkl8" Sep 16 18:48:15.379565 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:48:15.379530 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70580a6d_7e35_4eb0_85eb_65239da83d0b.slice/crio-278552aec44d0cb2a381287f0170807277a74637afc5d6b650259185d188ec04 WatchSource:0}: Error finding container 278552aec44d0cb2a381287f0170807277a74637afc5d6b650259185d188ec04: Status 404 returned error can't find the container with id 278552aec44d0cb2a381287f0170807277a74637afc5d6b650259185d188ec04 Sep 16 18:48:15.670672 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:48:15.670635 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40efaf55_090f_49b5_97fb_c3a3fef1e69f.slice/crio-5ca1cd0315faf48cf63b06dde3c2bc574cd102f034130315c4638603f501cce4 WatchSource:0}: Error finding container 5ca1cd0315faf48cf63b06dde3c2bc574cd102f034130315c4638603f501cce4: Status 404 returned error can't find the container with id 5ca1cd0315faf48cf63b06dde3c2bc574cd102f034130315c4638603f501cce4 Sep 16 18:48:15.673998 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:15.673806 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb"] Sep 16 18:48:15.895191 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:15.895153 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pdkl8" event={"ID":"70580a6d-7e35-4eb0-85eb-65239da83d0b","Type":"ContainerStarted","Data":"278552aec44d0cb2a381287f0170807277a74637afc5d6b650259185d188ec04"} Sep 16 18:48:15.896375 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:15.896349 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" event={"ID":"40efaf55-090f-49b5-97fb-c3a3fef1e69f","Type":"ContainerStarted","Data":"5ca1cd0315faf48cf63b06dde3c2bc574cd102f034130315c4638603f501cce4"} Sep 16 18:48:16.811665 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.811626 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-766d64ccfc-wzr59"] Sep 16 18:48:16.815996 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.815975 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.820782 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.820753 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4lgqmdd6182ig\"" Sep 16 18:48:16.820906 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.820809 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 16 18:48:16.820906 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.820854 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-54hg8\"" Sep 16 18:48:16.820906 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.820872 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 16 18:48:16.821214 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.821198 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 16 18:48:16.821414 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.821399 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 16 18:48:16.821482 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.821434 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 16 18:48:16.841493 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.841467 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-766d64ccfc-wzr59"] Sep 16 18:48:16.841869 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.841845 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.841986 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.841893 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-grpc-tls\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.841986 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.841924 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.841986 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.841955 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c907ae1-316c-4570-b5de-181d7dd64b61-metrics-client-ca\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.842141 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.842053 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.842141 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.842094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-tls\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.842141 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.842125 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.842255 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.842189 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4prmp\" (UniqueName: \"kubernetes.io/projected/0c907ae1-316c-4570-b5de-181d7dd64b61-kube-api-access-4prmp\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.943549 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.943503 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.943549 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.943552 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-grpc-tls\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.943766 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.943577 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.943857 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.943787 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c907ae1-316c-4570-b5de-181d7dd64b61-metrics-client-ca\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.943921 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.943860 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.943921 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.943901 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-tls\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.944019 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.943933 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.944019 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.943986 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4prmp\" (UniqueName: \"kubernetes.io/projected/0c907ae1-316c-4570-b5de-181d7dd64b61-kube-api-access-4prmp\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.944686 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.944660 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0c907ae1-316c-4570-b5de-181d7dd64b61-metrics-client-ca\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.946826 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.946804 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.947026 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.947006 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.947143 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.947126 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.947280 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.947229 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-tls\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.947417 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.947390 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.947769 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.947747 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0c907ae1-316c-4570-b5de-181d7dd64b61-secret-grpc-tls\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:16.952935 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:16.952915 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4prmp\" (UniqueName: \"kubernetes.io/projected/0c907ae1-316c-4570-b5de-181d7dd64b61-kube-api-access-4prmp\") pod \"thanos-querier-766d64ccfc-wzr59\" (UID: \"0c907ae1-316c-4570-b5de-181d7dd64b61\") " pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:17.126721 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:17.126687 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:17.439884 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:17.439802 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-766d64ccfc-wzr59"] Sep 16 18:48:17.444525 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:48:17.444498 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c907ae1_316c_4570_b5de_181d7dd64b61.slice/crio-c6202857a71d638440225898d3f1e6dd14dfc642ad64087b50aaf9984147bb92 WatchSource:0}: Error finding container c6202857a71d638440225898d3f1e6dd14dfc642ad64087b50aaf9984147bb92: Status 404 returned error can't find the container with id c6202857a71d638440225898d3f1e6dd14dfc642ad64087b50aaf9984147bb92 Sep 16 18:48:17.534386 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:48:17.534342 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-xrfb7" podUID="b1ceca87-379c-489d-b13f-6365d0a612b2" Sep 16 18:48:17.907480 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:17.907438 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" event={"ID":"0c907ae1-316c-4570-b5de-181d7dd64b61","Type":"ContainerStarted","Data":"c6202857a71d638440225898d3f1e6dd14dfc642ad64087b50aaf9984147bb92"} Sep 16 18:48:17.909460 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:17.909430 2563 generic.go:356] "Generic (PLEG): container finished" podID="70580a6d-7e35-4eb0-85eb-65239da83d0b" containerID="8c40553af9ea7522552df8b87cf970e6c4cfb93fb5e1301e2a9035bb5f5d132e" exitCode=0 Sep 16 18:48:17.909664 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:17.909517 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pdkl8" event={"ID":"70580a6d-7e35-4eb0-85eb-65239da83d0b","Type":"ContainerDied","Data":"8c40553af9ea7522552df8b87cf970e6c4cfb93fb5e1301e2a9035bb5f5d132e"} Sep 16 18:48:17.912312 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:17.912169 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" event={"ID":"40efaf55-090f-49b5-97fb-c3a3fef1e69f","Type":"ContainerStarted","Data":"f62e12f03d0226e68c70090e57cb1d180d1bee58b55717e70ad5af75922aea08"} Sep 16 18:48:17.912312 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:17.912242 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" event={"ID":"40efaf55-090f-49b5-97fb-c3a3fef1e69f","Type":"ContainerStarted","Data":"aa5caa813f2fcc839914abe4a60a168565f9915222e74aafe7d666b16ec834fb"} Sep 16 18:48:17.912312 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:17.912182 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xrfb7" Sep 16 18:48:17.912483 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:17.912280 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" event={"ID":"40efaf55-090f-49b5-97fb-c3a3fef1e69f","Type":"ContainerStarted","Data":"8c39f11e4a043a5ff7b3ff0c6fbcddfc71b2ff0ae6017acf3e6a57c2d72133d5"} Sep 16 18:48:17.955279 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:17.955107 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-k6mjb" podStartSLOduration=2.6052379119999998 podStartE2EDuration="3.955086473s" totalImagesPullingTime="1.349848561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:17.953500426 +0000 UTC m=+158.277193289" watchObservedRunningTime="2026-09-16 18:48:17.955086473 +0000 UTC m=+158.278779323" Sep 16 18:48:18.920263 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:18.920225 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pdkl8" event={"ID":"70580a6d-7e35-4eb0-85eb-65239da83d0b","Type":"ContainerStarted","Data":"050f1f92e9379887414897a5757778b2c1817c60555f34f2877dba9a5ca95056"} Sep 16 18:48:18.920761 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:18.920294 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pdkl8" event={"ID":"70580a6d-7e35-4eb0-85eb-65239da83d0b","Type":"ContainerStarted","Data":"44f4afce7d021560dd60cc638c5164067a240036746fbd347003ed27912f472c"} Sep 16 18:48:18.951938 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:18.951862 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pdkl8" podStartSLOduration=3.224375066 podStartE2EDuration="4.951826367s" totalImagesPullingTime="1.640034454s" totalInitContainerRuntime="87.416847ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:18.951132965 +0000 UTC m=+159.274825845" watchObservedRunningTime="2026-09-16 18:48:18.951826367 +0000 UTC m=+159.275519217" Sep 16 18:48:19.365636 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:48:19.365589 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-lq68v" podUID="650deff9-b845-4ef3-acad-6bdee9667025" Sep 16 18:48:19.926272 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:19.926228 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" event={"ID":"0c907ae1-316c-4570-b5de-181d7dd64b61","Type":"ContainerStarted","Data":"121ab8edc781b4cb6073369d63e9e160f7b778a3b26f83fbb4e427cfb9ad8b36"} Sep 16 18:48:19.926769 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:19.926325 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" event={"ID":"0c907ae1-316c-4570-b5de-181d7dd64b61","Type":"ContainerStarted","Data":"f7250e0d6006b3e91c06f8e48b3959b4b21e1dd39e879850de6aca3fc5ae7ef9"} Sep 16 18:48:19.926769 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:19.926353 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" event={"ID":"0c907ae1-316c-4570-b5de-181d7dd64b61","Type":"ContainerStarted","Data":"5ff6bd81c9a99eb56bf76b5e037b9677160c2bf32302e6019b999935172b47ac"} Sep 16 18:48:20.912809 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.912770 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 18:48:20.917172 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.917143 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.919994 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.919961 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lq7rp\"" Sep 16 18:48:20.920380 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.920360 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 16 18:48:20.920669 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.920650 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-hoqmcpnd0jd9\"" Sep 16 18:48:20.920905 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.920887 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 18:48:20.921147 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.921132 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 16 18:48:20.921746 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.921518 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 16 18:48:20.921746 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.921584 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 16 18:48:20.921746 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.921624 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 16 18:48:20.921746 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.921733 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 16 18:48:20.922118 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.921782 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 16 18:48:20.922118 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.922113 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 16 18:48:20.922334 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.922318 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 16 18:48:20.922492 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.922323 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 16 18:48:20.922674 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.922649 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 16 18:48:20.922776 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.922653 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 16 18:48:20.923223 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.923189 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 16 18:48:20.923758 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.923737 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 16 18:48:20.934016 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.933974 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" event={"ID":"0c907ae1-316c-4570-b5de-181d7dd64b61","Type":"ContainerStarted","Data":"d1d3f611e9145083316be41c7d88a1024fc7248423e7396cab2bf9779e835bbd"} Sep 16 18:48:20.934413 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.934093 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" event={"ID":"0c907ae1-316c-4570-b5de-181d7dd64b61","Type":"ContainerStarted","Data":"931f23a128e1e71cec8e028da20ac7f6eadbd455c56de207cb5f9155e1c2229a"} Sep 16 18:48:20.934413 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.934235 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:20.934413 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.934321 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" event={"ID":"0c907ae1-316c-4570-b5de-181d7dd64b61","Type":"ContainerStarted","Data":"8231f0db475dbcd2d0f28c85a2a37a7e7170f382ade510a9326bd8c1d7258a66"} Sep 16 18:48:20.946050 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.946013 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 18:48:20.985947 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.985897 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986096 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.985965 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-config-out\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986096 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986014 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986096 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986056 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986096 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986077 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986096 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986092 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986279 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986132 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986279 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986159 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986279 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986218 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986365 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986296 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986365 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986329 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986427 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986362 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986481 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986461 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-config\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986535 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986495 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986535 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986528 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986627 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986596 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986685 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986632 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-web-config\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986885 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986862 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986945 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986898 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:20.986945 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:20.986935 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qkn\" (UniqueName: \"kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-kube-api-access-47qkn\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.009883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.009801 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" podStartSLOduration=2.220721063 podStartE2EDuration="5.009782737s" totalImagesPullingTime="2.789061674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:21.007939419 +0000 UTC m=+161.331632268" watchObservedRunningTime="2026-09-16 18:48:21.009782737 +0000 UTC m=+161.333475586" Sep 16 18:48:21.087650 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.087615 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.087650 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.087653 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.087961 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.087672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.087961 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.087792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.087961 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.087827 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.087961 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.087877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.087961 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.087934 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-config\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.087961 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.087956 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088245 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.087985 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088245 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.088013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088245 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.088037 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-web-config\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088245 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.088077 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088245 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.088101 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088245 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.088124 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qkn\" (UniqueName: \"kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-kube-api-access-47qkn\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088245 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.088199 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088245 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.088238 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-config-out\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088602 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.088268 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088602 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.088294 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088602 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.088322 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.088602 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.088347 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.089650 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.089441 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.090773 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.090749 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.091343 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.091315 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.091781 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.091651 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.091781 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.091798 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-web-config\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.095458 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.092308 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.095458 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.092665 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.095458 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.092316 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.095458 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.093069 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.095458 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.093221 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.095458 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.093994 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.095458 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.094028 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-config-out\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.096027 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.095998 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.096133 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.096070 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.096199 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.096164 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.096199 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.096163 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.096773 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.096627 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-config\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.096773 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.096748 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.098130 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.098104 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.122785 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.122742 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qkn\" (UniqueName: \"kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-kube-api-access-47qkn\") pod \"prometheus-k8s-0\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.231280 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.231174 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:21.522312 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.522275 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 18:48:21.525510 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:48:21.525481 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6afb9003_2a50_42e5_b25e_aa5072f19810.slice/crio-77832e0a2e57dbe75dbf4b35a46baeb32e8de23ee7f2d2c1e5a5530933c153ca WatchSource:0}: Error finding container 77832e0a2e57dbe75dbf4b35a46baeb32e8de23ee7f2d2c1e5a5530933c153ca: Status 404 returned error can't find the container with id 77832e0a2e57dbe75dbf4b35a46baeb32e8de23ee7f2d2c1e5a5530933c153ca Sep 16 18:48:21.940152 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:21.940057 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerStarted","Data":"77832e0a2e57dbe75dbf4b35a46baeb32e8de23ee7f2d2c1e5a5530933c153ca"} Sep 16 18:48:22.402888 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:22.402846 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:48:22.405100 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:22.405069 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b1ceca87-379c-489d-b13f-6365d0a612b2-metrics-tls\") pod \"dns-default-xrfb7\" (UID: \"b1ceca87-379c-489d-b13f-6365d0a612b2\") " pod="openshift-dns/dns-default-xrfb7" Sep 16 18:48:22.416432 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:22.416401 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qgt44\"" Sep 16 18:48:22.423630 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:22.423614 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xrfb7" Sep 16 18:48:22.723855 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:22.723747 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xrfb7"] Sep 16 18:48:22.727455 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:48:22.727424 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ceca87_379c_489d_b13f_6365d0a612b2.slice/crio-63ca06d15f843266db4b77dbd1c91595e81bfa4c5e04cf6aa2098b17b345cc39 WatchSource:0}: Error finding container 63ca06d15f843266db4b77dbd1c91595e81bfa4c5e04cf6aa2098b17b345cc39: Status 404 returned error can't find the container with id 63ca06d15f843266db4b77dbd1c91595e81bfa4c5e04cf6aa2098b17b345cc39 Sep 16 18:48:22.946124 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:22.946090 2563 generic.go:356] "Generic (PLEG): container finished" podID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerID="f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5" exitCode=0 Sep 16 18:48:22.946563 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:22.946170 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerDied","Data":"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5"} Sep 16 18:48:22.947633 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:22.947607 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xrfb7" event={"ID":"b1ceca87-379c-489d-b13f-6365d0a612b2","Type":"ContainerStarted","Data":"63ca06d15f843266db4b77dbd1c91595e81bfa4c5e04cf6aa2098b17b345cc39"} Sep 16 18:48:24.960748 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:24.960709 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xrfb7" event={"ID":"b1ceca87-379c-489d-b13f-6365d0a612b2","Type":"ContainerStarted","Data":"e25cf4080e0f097d7709b0b450a9ad5284bcdbdaff81a15c53e7b5129a864e1d"} Sep 16 18:48:24.961366 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:24.960771 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xrfb7" event={"ID":"b1ceca87-379c-489d-b13f-6365d0a612b2","Type":"ContainerStarted","Data":"3292da6e0454a72b2249e41e74297149d47495a77459ad1ed08c3524dc7c6125"} Sep 16 18:48:24.961366 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:24.960863 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xrfb7" Sep 16 18:48:24.989214 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:24.989151 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xrfb7" podStartSLOduration=129.372682492 podStartE2EDuration="2m10.989131286s" totalImagesPullingTime="1.616448794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:24.988521901 +0000 UTC m=+165.312214750" watchObservedRunningTime="2026-09-16 18:48:24.989131286 +0000 UTC m=+165.312824191" Sep 16 18:48:26.946185 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:26.946156 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-766d64ccfc-wzr59" Sep 16 18:48:26.974508 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:26.974243 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerStarted","Data":"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9"} Sep 16 18:48:26.974508 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:26.974330 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerStarted","Data":"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030"} Sep 16 18:48:26.974508 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:26.974370 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerStarted","Data":"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4"} Sep 16 18:48:26.974508 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:26.974406 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerStarted","Data":"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad"} Sep 16 18:48:26.974508 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:26.974445 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerStarted","Data":"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378"} Sep 16 18:48:26.974508 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:26.974482 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerStarted","Data":"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142"} Sep 16 18:48:27.018493 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:27.018412 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.980256777 podStartE2EDuration="7.018390161s" totalImagesPullingTime="4.015577023s" totalInitContainerRuntime="22.556361ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:48:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:27.016715615 +0000 UTC m=+167.340408464" watchObservedRunningTime="2026-09-16 18:48:27.018390161 +0000 UTC m=+167.342083011" Sep 16 18:48:27.331527 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:27.331484 2563 scope.go:117] "RemoveContainer" containerID="d1b4a70d71d40d0d60d1c41f454fae1f8250adbd3029c79e2d6cdeb510c44a59" Sep 16 18:48:27.983595 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:27.983563 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 18:48:27.984087 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:27.983650 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" event={"ID":"322e4d63-e09c-4b27-8681-32b11e13c0cf","Type":"ContainerStarted","Data":"0ebc35377f7f6aed154462141ec543df2c26fee309bda8ec5e86443814797ccf"} Sep 16 18:48:27.984511 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:27.984482 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:48:27.990149 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:27.990125 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" Sep 16 18:48:28.010525 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:28.010470 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-f2hw9" podStartSLOduration=52.60179204 podStartE2EDuration="55.010451887s" totalImagesPullingTime="2.408659847s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:48:28.009079203 +0000 UTC m=+168.332772050" watchObservedRunningTime="2026-09-16 18:48:28.010451887 +0000 UTC m=+168.334144735" Sep 16 18:48:31.231997 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:31.231955 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:48:31.331001 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:31.330959 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:48:34.968509 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:34.968427 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xrfb7" Sep 16 18:48:47.062314 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:47.062274 2563 generic.go:356] "Generic (PLEG): container finished" podID="492c472f-5f87-465e-9534-058f606e5944" containerID="5bb070caacd446ac06515faa9fca8a2bcca1c2866ac3a005ba547824e7fa5183" exitCode=0 Sep 16 18:48:47.062796 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:47.062354 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" event={"ID":"492c472f-5f87-465e-9534-058f606e5944","Type":"ContainerDied","Data":"5bb070caacd446ac06515faa9fca8a2bcca1c2866ac3a005ba547824e7fa5183"} Sep 16 18:48:47.062796 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:47.062784 2563 scope.go:117] "RemoveContainer" containerID="5bb070caacd446ac06515faa9fca8a2bcca1c2866ac3a005ba547824e7fa5183" Sep 16 18:48:48.068436 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:48.068399 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4rpwl" event={"ID":"492c472f-5f87-465e-9534-058f606e5944","Type":"ContainerStarted","Data":"d4dc2fb0439b818ff6d3eb52370f128d5919b38d2fb7ff422a9a9cdb10f90f18"} Sep 16 18:48:49.039668 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:49.039638 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-hms5p_4f1e811d-0e94-42b6-888c-3e2f1952434c/cluster-monitoring-operator/0.log" Sep 16 18:48:49.237016 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:49.236984 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-k6mjb_40efaf55-090f-49b5-97fb-c3a3fef1e69f/kube-state-metrics/0.log" Sep 16 18:48:49.436293 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:49.436201 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-k6mjb_40efaf55-090f-49b5-97fb-c3a3fef1e69f/kube-rbac-proxy-main/0.log" Sep 16 18:48:49.635593 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:49.635557 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-k6mjb_40efaf55-090f-49b5-97fb-c3a3fef1e69f/kube-rbac-proxy-self/0.log" Sep 16 18:48:50.836898 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:50.836867 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pdkl8_70580a6d-7e35-4eb0-85eb-65239da83d0b/init-textfile/0.log" Sep 16 18:48:51.038904 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:51.038875 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pdkl8_70580a6d-7e35-4eb0-85eb-65239da83d0b/node-exporter/0.log" Sep 16 18:48:51.237196 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:51.237118 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pdkl8_70580a6d-7e35-4eb0-85eb-65239da83d0b/kube-rbac-proxy/0.log" Sep 16 18:48:52.639189 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:52.639134 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6afb9003-2a50-42e5-b25e-aa5072f19810/init-config-reloader/0.log" Sep 16 18:48:52.838304 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:52.838240 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6afb9003-2a50-42e5-b25e-aa5072f19810/prometheus/0.log" Sep 16 18:48:53.037294 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:53.037245 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6afb9003-2a50-42e5-b25e-aa5072f19810/config-reloader/0.log" Sep 16 18:48:53.237193 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:53.237163 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6afb9003-2a50-42e5-b25e-aa5072f19810/thanos-sidecar/0.log" Sep 16 18:48:53.438323 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:53.438247 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6afb9003-2a50-42e5-b25e-aa5072f19810/kube-rbac-proxy-web/0.log" Sep 16 18:48:53.636911 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:53.636883 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6afb9003-2a50-42e5-b25e-aa5072f19810/kube-rbac-proxy/0.log" Sep 16 18:48:53.835596 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:53.835516 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6afb9003-2a50-42e5-b25e-aa5072f19810/kube-rbac-proxy-thanos/0.log" Sep 16 18:48:54.036694 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:54.036657 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-xgwks_a47d1919-84c7-4649-af42-ed5e2c37c05a/prometheus-operator/0.log" Sep 16 18:48:54.237196 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:54.237089 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-xgwks_a47d1919-84c7-4649-af42-ed5e2c37c05a/kube-rbac-proxy/0.log" Sep 16 18:48:54.640758 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:54.640729 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/thanos-query/0.log" Sep 16 18:48:54.836017 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:54.835976 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/kube-rbac-proxy-web/0.log" Sep 16 18:48:55.035701 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:55.035672 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/kube-rbac-proxy/0.log" Sep 16 18:48:55.235445 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:55.235417 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/prom-label-proxy/0.log" Sep 16 18:48:55.435716 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:55.435597 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/kube-rbac-proxy-rules/0.log" Sep 16 18:48:55.635456 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:55.635400 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/kube-rbac-proxy-metrics/0.log" Sep 16 18:48:55.835331 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:55.835283 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-nqzzv_7b5367e2-4fb0-43ec-8046-f1fdf490abeb/networking-console-plugin/0.log" Sep 16 18:48:56.035862 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:56.035819 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 18:48:56.237666 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:48:56.237591 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/3.log" Sep 16 18:49:21.232155 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:21.232119 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:21.249145 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:21.249115 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:22.233505 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:22.233470 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:39.346342 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.346299 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 18:49:39.346989 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.346651 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="thanos-sidecar" containerID="cri-o://c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad" gracePeriod=600 Sep 16 18:49:39.346989 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.346668 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy-thanos" containerID="cri-o://98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9" gracePeriod=600 Sep 16 18:49:39.346989 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.346692 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="config-reloader" containerID="cri-o://6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378" gracePeriod=600 Sep 16 18:49:39.346989 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.346667 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy-web" containerID="cri-o://026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4" gracePeriod=600 Sep 16 18:49:39.346989 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.346637 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy" containerID="cri-o://ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030" gracePeriod=600 Sep 16 18:49:39.346989 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.346596 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="prometheus" containerID="cri-o://d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142" gracePeriod=600 Sep 16 18:49:39.591021 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.590126 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:39.614784 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.614696 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-db\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.614784 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.614740 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-trusted-ca-bundle\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.614784 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.614776 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-config-out\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615076 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.614803 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-thanos-prometheus-http-client-file\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615076 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.614849 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-tls\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615076 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.614885 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-kube-rbac-proxy\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615076 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.614914 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-metrics-client-certs\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615076 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.614942 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-grpc-tls\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615076 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.614970 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-1\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615076 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615014 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-metrics-client-ca\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615076 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615042 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615076 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615069 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-0\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615572 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615090 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-config\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615572 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615146 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-tls-assets\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615572 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615168 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-web-config\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615572 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615197 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-serving-certs-ca-bundle\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615572 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615232 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615572 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615275 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47qkn\" (UniqueName: \"kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-kube-api-access-47qkn\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615572 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615307 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-2\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615572 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615331 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-kubelet-serving-ca-bundle\") pod \"6afb9003-2a50-42e5-b25e-aa5072f19810\" (UID: \"6afb9003-2a50-42e5-b25e-aa5072f19810\") " Sep 16 18:49:39.615982 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.615905 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:39.617044 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.616412 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:49:39.617044 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.616423 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:39.617044 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.616779 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:39.617044 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.616787 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:39.617560 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.617529 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:39.618029 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.617999 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:39.621859 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.621177 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:39.623739 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.622196 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:49:39.623739 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.623226 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:39.623739 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.623306 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-kube-api-access-47qkn" (OuterVolumeSpecName: "kube-api-access-47qkn") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "kube-api-access-47qkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:49:39.623739 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.623478 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-config" (OuterVolumeSpecName: "config") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:39.623739 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.623697 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:39.623739 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.623722 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:39.624255 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.624219 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:39.624356 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.624262 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:39.624498 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.624437 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:39.624608 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.624537 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-config-out" (OuterVolumeSpecName: "config-out") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:49:39.626083 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.626053 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:49:39.639648 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.639596 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-web-config" (OuterVolumeSpecName: "web-config") pod "6afb9003-2a50-42e5-b25e-aa5072f19810" (UID: "6afb9003-2a50-42e5-b25e-aa5072f19810"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:49:39.716187 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716150 2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-metrics-client-ca\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716187 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716188 2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716202 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716211 2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-config\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716219 2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-tls-assets\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716229 2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-web-config\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716236 2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716244 2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716253 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47qkn\" (UniqueName: \"kubernetes.io/projected/6afb9003-2a50-42e5-b25e-aa5072f19810-kube-api-access-47qkn\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716265 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716279 2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716294 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-db\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716304 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716312 2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6afb9003-2a50-42e5-b25e-aa5072f19810-config-out\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716321 2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716329 2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716340 2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-kube-rbac-proxy\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716349 2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-metrics-client-certs\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716357 2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6afb9003-2a50-42e5-b25e-aa5072f19810-secret-grpc-tls\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:39.716378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:39.716365 2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6afb9003-2a50-42e5-b25e-aa5072f19810-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:49:40.295533 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295494 2563 generic.go:356] "Generic (PLEG): container finished" podID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerID="98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9" exitCode=0 Sep 16 18:49:40.295533 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295528 2563 generic.go:356] "Generic (PLEG): container finished" podID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerID="ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030" exitCode=0 Sep 16 18:49:40.295533 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295537 2563 generic.go:356] "Generic (PLEG): container finished" podID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerID="026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4" exitCode=0 Sep 16 18:49:40.295762 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295546 2563 generic.go:356] "Generic (PLEG): container finished" podID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerID="c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad" exitCode=0 Sep 16 18:49:40.295762 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295554 2563 generic.go:356] "Generic (PLEG): container finished" podID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerID="6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378" exitCode=0 Sep 16 18:49:40.295762 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295562 2563 generic.go:356] "Generic (PLEG): container finished" podID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerID="d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142" exitCode=0 Sep 16 18:49:40.295762 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295563 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerDied","Data":"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9"} Sep 16 18:49:40.295762 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295597 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.295762 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295652 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerDied","Data":"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030"} Sep 16 18:49:40.295762 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295681 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerDied","Data":"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4"} Sep 16 18:49:40.295762 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295707 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerDied","Data":"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad"} Sep 16 18:49:40.295762 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295726 2563 scope.go:117] "RemoveContainer" containerID="98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9" Sep 16 18:49:40.295762 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295732 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerDied","Data":"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378"} Sep 16 18:49:40.295762 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295756 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerDied","Data":"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142"} Sep 16 18:49:40.296179 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.295779 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6afb9003-2a50-42e5-b25e-aa5072f19810","Type":"ContainerDied","Data":"77832e0a2e57dbe75dbf4b35a46baeb32e8de23ee7f2d2c1e5a5530933c153ca"} Sep 16 18:49:40.306168 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.306150 2563 scope.go:117] "RemoveContainer" containerID="ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030" Sep 16 18:49:40.312868 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.312827 2563 scope.go:117] "RemoveContainer" containerID="026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4" Sep 16 18:49:40.319126 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.319111 2563 scope.go:117] "RemoveContainer" containerID="c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad" Sep 16 18:49:40.323080 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.323061 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 18:49:40.326569 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.326486 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 18:49:40.327017 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.326996 2563 scope.go:117] "RemoveContainer" containerID="6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378" Sep 16 18:49:40.334903 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.334884 2563 scope.go:117] "RemoveContainer" containerID="d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142" Sep 16 18:49:40.337173 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.337150 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" path="/var/lib/kubelet/pods/6afb9003-2a50-42e5-b25e-aa5072f19810/volumes" Sep 16 18:49:40.342443 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.342423 2563 scope.go:117] "RemoveContainer" containerID="f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5" Sep 16 18:49:40.350042 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.349826 2563 scope.go:117] "RemoveContainer" containerID="98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9" Sep 16 18:49:40.350290 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:49:40.350208 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": container with ID starting with 98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9 not found: ID does not exist" containerID="98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9" Sep 16 18:49:40.350290 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.350238 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9"} err="failed to get container status \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": rpc error: code = NotFound desc = could not find container \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": container with ID starting with 98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9 not found: ID does not exist" Sep 16 18:49:40.350290 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.350271 2563 scope.go:117] "RemoveContainer" containerID="ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030" Sep 16 18:49:40.350547 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:49:40.350527 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": container with ID starting with ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030 not found: ID does not exist" containerID="ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030" Sep 16 18:49:40.350590 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.350553 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030"} err="failed to get container status \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": rpc error: code = NotFound desc = could not find container \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": container with ID starting with ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030 not found: ID does not exist" Sep 16 18:49:40.350590 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.350570 2563 scope.go:117] "RemoveContainer" containerID="026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4" Sep 16 18:49:40.350824 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:49:40.350808 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": container with ID starting with 026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4 not found: ID does not exist" containerID="026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4" Sep 16 18:49:40.350909 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.350826 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4"} err="failed to get container status \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": rpc error: code = NotFound desc = could not find container \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": container with ID starting with 026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4 not found: ID does not exist" Sep 16 18:49:40.350909 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.350862 2563 scope.go:117] "RemoveContainer" containerID="c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad" Sep 16 18:49:40.351106 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:49:40.351092 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": container with ID starting with c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad not found: ID does not exist" containerID="c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad" Sep 16 18:49:40.351145 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.351109 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad"} err="failed to get container status \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": rpc error: code = NotFound desc = could not find container \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": container with ID starting with c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad not found: ID does not exist" Sep 16 18:49:40.351145 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.351120 2563 scope.go:117] "RemoveContainer" containerID="6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378" Sep 16 18:49:40.351354 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:49:40.351337 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": container with ID starting with 6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378 not found: ID does not exist" containerID="6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378" Sep 16 18:49:40.351404 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.351357 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378"} err="failed to get container status \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": rpc error: code = NotFound desc = could not find container \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": container with ID starting with 6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378 not found: ID does not exist" Sep 16 18:49:40.351404 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.351369 2563 scope.go:117] "RemoveContainer" containerID="d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142" Sep 16 18:49:40.351570 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:49:40.351555 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": container with ID starting with d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142 not found: ID does not exist" containerID="d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142" Sep 16 18:49:40.351611 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.351573 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142"} err="failed to get container status \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": rpc error: code = NotFound desc = could not find container \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": container with ID starting with d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142 not found: ID does not exist" Sep 16 18:49:40.351611 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.351584 2563 scope.go:117] "RemoveContainer" containerID="f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5" Sep 16 18:49:40.351789 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:49:40.351775 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": container with ID starting with f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5 not found: ID does not exist" containerID="f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5" Sep 16 18:49:40.351852 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.351792 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5"} err="failed to get container status \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": rpc error: code = NotFound desc = could not find container \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": container with ID starting with f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5 not found: ID does not exist" Sep 16 18:49:40.351852 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.351803 2563 scope.go:117] "RemoveContainer" containerID="98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9" Sep 16 18:49:40.352030 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.352014 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9"} err="failed to get container status \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": rpc error: code = NotFound desc = could not find container \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": container with ID starting with 98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9 not found: ID does not exist" Sep 16 18:49:40.352070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.352031 2563 scope.go:117] "RemoveContainer" containerID="ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030" Sep 16 18:49:40.352246 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.352229 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030"} err="failed to get container status \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": rpc error: code = NotFound desc = could not find container \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": container with ID starting with ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030 not found: ID does not exist" Sep 16 18:49:40.352246 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.352245 2563 scope.go:117] "RemoveContainer" containerID="026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4" Sep 16 18:49:40.352463 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.352447 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4"} err="failed to get container status \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": rpc error: code = NotFound desc = could not find container \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": container with ID starting with 026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4 not found: ID does not exist" Sep 16 18:49:40.352503 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.352463 2563 scope.go:117] "RemoveContainer" containerID="c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad" Sep 16 18:49:40.352695 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.352674 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad"} err="failed to get container status \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": rpc error: code = NotFound desc = could not find container \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": container with ID starting with c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad not found: ID does not exist" Sep 16 18:49:40.352745 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.352696 2563 scope.go:117] "RemoveContainer" containerID="6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378" Sep 16 18:49:40.352920 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.352906 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378"} err="failed to get container status \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": rpc error: code = NotFound desc = could not find container \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": container with ID starting with 6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378 not found: ID does not exist" Sep 16 18:49:40.352920 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.352919 2563 scope.go:117] "RemoveContainer" containerID="d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142" Sep 16 18:49:40.353128 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.353113 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142"} err="failed to get container status \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": rpc error: code = NotFound desc = could not find container \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": container with ID starting with d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142 not found: ID does not exist" Sep 16 18:49:40.353169 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.353127 2563 scope.go:117] "RemoveContainer" containerID="f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5" Sep 16 18:49:40.353322 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.353308 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5"} err="failed to get container status \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": rpc error: code = NotFound desc = could not find container \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": container with ID starting with f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5 not found: ID does not exist" Sep 16 18:49:40.353322 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.353320 2563 scope.go:117] "RemoveContainer" containerID="98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9" Sep 16 18:49:40.353520 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.353506 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9"} err="failed to get container status \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": rpc error: code = NotFound desc = could not find container \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": container with ID starting with 98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9 not found: ID does not exist" Sep 16 18:49:40.353520 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.353519 2563 scope.go:117] "RemoveContainer" containerID="ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030" Sep 16 18:49:40.353723 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.353703 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030"} err="failed to get container status \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": rpc error: code = NotFound desc = could not find container \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": container with ID starting with ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030 not found: ID does not exist" Sep 16 18:49:40.353842 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.353723 2563 scope.go:117] "RemoveContainer" containerID="026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4" Sep 16 18:49:40.354055 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.354033 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4"} err="failed to get container status \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": rpc error: code = NotFound desc = could not find container \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": container with ID starting with 026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4 not found: ID does not exist" Sep 16 18:49:40.354125 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.354056 2563 scope.go:117] "RemoveContainer" containerID="c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad" Sep 16 18:49:40.354350 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.354323 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad"} err="failed to get container status \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": rpc error: code = NotFound desc = could not find container \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": container with ID starting with c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad not found: ID does not exist" Sep 16 18:49:40.355069 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.354351 2563 scope.go:117] "RemoveContainer" containerID="6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378" Sep 16 18:49:40.355069 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.354657 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378"} err="failed to get container status \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": rpc error: code = NotFound desc = could not find container \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": container with ID starting with 6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378 not found: ID does not exist" Sep 16 18:49:40.355069 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.354676 2563 scope.go:117] "RemoveContainer" containerID="d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142" Sep 16 18:49:40.355069 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.355017 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142"} err="failed to get container status \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": rpc error: code = NotFound desc = could not find container \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": container with ID starting with d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142 not found: ID does not exist" Sep 16 18:49:40.355069 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.355039 2563 scope.go:117] "RemoveContainer" containerID="f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5" Sep 16 18:49:40.355351 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.355323 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5"} err="failed to get container status \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": rpc error: code = NotFound desc = could not find container \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": container with ID starting with f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5 not found: ID does not exist" Sep 16 18:49:40.355351 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.355344 2563 scope.go:117] "RemoveContainer" containerID="98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9" Sep 16 18:49:40.355649 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.355625 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9"} err="failed to get container status \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": rpc error: code = NotFound desc = could not find container \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": container with ID starting with 98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9 not found: ID does not exist" Sep 16 18:49:40.355706 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.355650 2563 scope.go:117] "RemoveContainer" containerID="ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030" Sep 16 18:49:40.355930 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.355911 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030"} err="failed to get container status \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": rpc error: code = NotFound desc = could not find container \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": container with ID starting with ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030 not found: ID does not exist" Sep 16 18:49:40.355930 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.355930 2563 scope.go:117] "RemoveContainer" containerID="026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4" Sep 16 18:49:40.356021 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.355979 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 18:49:40.356189 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356169 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4"} err="failed to get container status \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": rpc error: code = NotFound desc = could not find container \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": container with ID starting with 026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4 not found: ID does not exist" Sep 16 18:49:40.356228 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356191 2563 scope.go:117] "RemoveContainer" containerID="c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad" Sep 16 18:49:40.356346 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356332 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy-web" Sep 16 18:49:40.356380 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356348 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy-web" Sep 16 18:49:40.356380 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356357 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="init-config-reloader" Sep 16 18:49:40.356380 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356363 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="init-config-reloader" Sep 16 18:49:40.356380 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356376 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356385 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356404 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="thanos-sidecar" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356409 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="thanos-sidecar" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356416 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy-thanos" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356422 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy-thanos" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356429 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="prometheus" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356435 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="prometheus" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356440 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="config-reloader" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356446 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="config-reloader" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356400 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad"} err="failed to get container status \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": rpc error: code = NotFound desc = could not find container \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": container with ID starting with c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad not found: ID does not exist" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356487 2563 scope.go:117] "RemoveContainer" containerID="6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356493 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="thanos-sidecar" Sep 16 18:49:40.356501 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356503 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy-web" Sep 16 18:49:40.356956 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356508 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy" Sep 16 18:49:40.356956 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356514 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="kube-rbac-proxy-thanos" Sep 16 18:49:40.356956 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356522 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="config-reloader" Sep 16 18:49:40.356956 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356527 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6afb9003-2a50-42e5-b25e-aa5072f19810" containerName="prometheus" Sep 16 18:49:40.356956 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356688 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378"} err="failed to get container status \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": rpc error: code = NotFound desc = could not find container \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": container with ID starting with 6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378 not found: ID does not exist" Sep 16 18:49:40.356956 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356705 2563 scope.go:117] "RemoveContainer" containerID="d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142" Sep 16 18:49:40.356956 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356951 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142"} err="failed to get container status \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": rpc error: code = NotFound desc = could not find container \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": container with ID starting with d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142 not found: ID does not exist" Sep 16 18:49:40.357262 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.356965 2563 scope.go:117] "RemoveContainer" containerID="f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5" Sep 16 18:49:40.357262 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.357176 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5"} err="failed to get container status \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": rpc error: code = NotFound desc = could not find container \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": container with ID starting with f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5 not found: ID does not exist" Sep 16 18:49:40.357262 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.357189 2563 scope.go:117] "RemoveContainer" containerID="98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9" Sep 16 18:49:40.357440 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.357422 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9"} err="failed to get container status \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": rpc error: code = NotFound desc = could not find container \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": container with ID starting with 98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9 not found: ID does not exist" Sep 16 18:49:40.357476 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.357440 2563 scope.go:117] "RemoveContainer" containerID="ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030" Sep 16 18:49:40.357653 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.357634 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030"} err="failed to get container status \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": rpc error: code = NotFound desc = could not find container \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": container with ID starting with ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030 not found: ID does not exist" Sep 16 18:49:40.357693 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.357653 2563 scope.go:117] "RemoveContainer" containerID="026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4" Sep 16 18:49:40.357886 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.357868 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4"} err="failed to get container status \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": rpc error: code = NotFound desc = could not find container \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": container with ID starting with 026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4 not found: ID does not exist" Sep 16 18:49:40.357886 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.357884 2563 scope.go:117] "RemoveContainer" containerID="c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad" Sep 16 18:49:40.358128 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.358110 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad"} err="failed to get container status \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": rpc error: code = NotFound desc = could not find container \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": container with ID starting with c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad not found: ID does not exist" Sep 16 18:49:40.358173 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.358129 2563 scope.go:117] "RemoveContainer" containerID="6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378" Sep 16 18:49:40.358343 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.358327 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378"} err="failed to get container status \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": rpc error: code = NotFound desc = could not find container \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": container with ID starting with 6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378 not found: ID does not exist" Sep 16 18:49:40.358383 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.358342 2563 scope.go:117] "RemoveContainer" containerID="d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142" Sep 16 18:49:40.358527 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.358512 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142"} err="failed to get container status \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": rpc error: code = NotFound desc = could not find container \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": container with ID starting with d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142 not found: ID does not exist" Sep 16 18:49:40.358566 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.358527 2563 scope.go:117] "RemoveContainer" containerID="f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5" Sep 16 18:49:40.358820 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.358796 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5"} err="failed to get container status \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": rpc error: code = NotFound desc = could not find container \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": container with ID starting with f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5 not found: ID does not exist" Sep 16 18:49:40.358895 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.358820 2563 scope.go:117] "RemoveContainer" containerID="98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9" Sep 16 18:49:40.359199 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.359178 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9"} err="failed to get container status \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": rpc error: code = NotFound desc = could not find container \"98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9\": container with ID starting with 98b282d5172e6058fd935242c70661166c9302f4cf59e0f00462e4d39555b8b9 not found: ID does not exist" Sep 16 18:49:40.359277 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.359201 2563 scope.go:117] "RemoveContainer" containerID="ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030" Sep 16 18:49:40.359451 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.359434 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030"} err="failed to get container status \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": rpc error: code = NotFound desc = could not find container \"ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030\": container with ID starting with ab991577831cb999e3ae510bb7fb735e7754843e03b4742a4603a7637955e030 not found: ID does not exist" Sep 16 18:49:40.359451 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.359451 2563 scope.go:117] "RemoveContainer" containerID="026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4" Sep 16 18:49:40.359683 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.359667 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4"} err="failed to get container status \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": rpc error: code = NotFound desc = could not find container \"026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4\": container with ID starting with 026caf4abb8e2bab52eba15a41d693be7c0d747127f33261df6163e0381b82e4 not found: ID does not exist" Sep 16 18:49:40.359683 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.359682 2563 scope.go:117] "RemoveContainer" containerID="c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad" Sep 16 18:49:40.359954 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.359937 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad"} err="failed to get container status \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": rpc error: code = NotFound desc = could not find container \"c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad\": container with ID starting with c0c6bacd15b97bd16957b1a03e59c36f5a044669082f70b9296f3593b5df36ad not found: ID does not exist" Sep 16 18:49:40.360009 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.359956 2563 scope.go:117] "RemoveContainer" containerID="6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378" Sep 16 18:49:40.360218 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.360197 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378"} err="failed to get container status \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": rpc error: code = NotFound desc = could not find container \"6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378\": container with ID starting with 6602f8e87e2a9425a0d2dbd1c8fd79d6dd1d020161f453e1f7d29634451a3378 not found: ID does not exist" Sep 16 18:49:40.360218 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.360215 2563 scope.go:117] "RemoveContainer" containerID="d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142" Sep 16 18:49:40.360490 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.360471 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142"} err="failed to get container status \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": rpc error: code = NotFound desc = could not find container \"d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142\": container with ID starting with d08611ab492d0b817cd245d6da7d3d238ba6e68569ae2eda4acf619b43160142 not found: ID does not exist" Sep 16 18:49:40.360490 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.360490 2563 scope.go:117] "RemoveContainer" containerID="f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5" Sep 16 18:49:40.360743 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.360722 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5"} err="failed to get container status \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": rpc error: code = NotFound desc = could not find container \"f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5\": container with ID starting with f64a0fae08acc690c748b195c0667d8b2b172ba592f24f1e947ebac28696eef5 not found: ID does not exist" Sep 16 18:49:40.362346 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.362329 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.364856 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.364754 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 16 18:49:40.364942 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.364860 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 16 18:49:40.364995 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.364944 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 16 18:49:40.364995 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.364909 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 16 18:49:40.365115 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.365099 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-hoqmcpnd0jd9\"" Sep 16 18:49:40.365318 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.365288 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 16 18:49:40.365318 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.365317 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 16 18:49:40.367878 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.367283 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 16 18:49:40.367878 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.367358 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lq7rp\"" Sep 16 18:49:40.367878 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.367585 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 16 18:49:40.367878 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.367607 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 16 18:49:40.367878 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.367660 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 16 18:49:40.367878 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.367607 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 16 18:49:40.367878 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.367776 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 16 18:49:40.368234 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.368128 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 18:49:40.369052 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.369029 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 16 18:49:40.372378 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.372360 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 16 18:49:40.375609 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.375592 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 18:49:40.422577 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.422553 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.422668 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.422588 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.422668 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.422603 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.422668 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.422628 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc7x7\" (UniqueName: \"kubernetes.io/projected/92d7889a-6c53-4354-8c2e-76f4b64c20b3-kube-api-access-nc7x7\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.422668 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.422660 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.422806 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.422742 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.422853 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.422843 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92d7889a-6c53-4354-8c2e-76f4b64c20b3-config-out\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.422889 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.422865 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.422889 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.422884 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.422956 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.422906 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.422989 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.422977 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.423031 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.423013 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.423063 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.423044 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.423094 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.423070 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.423127 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.423109 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.423159 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.423135 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92d7889a-6c53-4354-8c2e-76f4b64c20b3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.423212 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.423196 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.423242 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.423221 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-config\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.423274 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.423242 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-web-config\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.423333 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.423313 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.523925 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.523889 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92d7889a-6c53-4354-8c2e-76f4b64c20b3-config-out\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.523925 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.523920 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524102 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.523938 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524102 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524048 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524183 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524183 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524127 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524183 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524157 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524393 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524182 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524393 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524208 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524393 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524251 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92d7889a-6c53-4354-8c2e-76f4b64c20b3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524393 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524299 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524393 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524324 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-config\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524393 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524363 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-web-config\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524711 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524398 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524711 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524425 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524711 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524451 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524711 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524475 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524711 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524503 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc7x7\" (UniqueName: \"kubernetes.io/projected/92d7889a-6c53-4354-8c2e-76f4b64c20b3-kube-api-access-nc7x7\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524711 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.524711 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524579 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.525078 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524805 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.525078 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.524815 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.526430 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.525502 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.526430 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.525862 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.526430 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.526356 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.526981 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.526948 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.527258 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.527060 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/92d7889a-6c53-4354-8c2e-76f4b64c20b3-config-out\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.527466 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.527442 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.527549 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.527446 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.528142 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.528117 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-web-config\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.528727 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.528625 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.529032 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.529008 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.529121 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.529108 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-config\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.529458 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.529433 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.529458 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.529445 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/92d7889a-6c53-4354-8c2e-76f4b64c20b3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.529719 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.529700 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/92d7889a-6c53-4354-8c2e-76f4b64c20b3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.530255 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.530237 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.530316 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.530267 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.530954 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.530934 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/92d7889a-6c53-4354-8c2e-76f4b64c20b3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.534181 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.534165 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc7x7\" (UniqueName: \"kubernetes.io/projected/92d7889a-6c53-4354-8c2e-76f4b64c20b3-kube-api-access-nc7x7\") pod \"prometheus-k8s-0\" (UID: \"92d7889a-6c53-4354-8c2e-76f4b64c20b3\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.684666 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.684584 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:40.970527 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:40.970451 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 18:49:40.973794 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:49:40.973767 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d7889a_6c53_4354_8c2e_76f4b64c20b3.slice/crio-c5d43d150263b621d28465ef884b042972cefa6eea75fa4c8eb8ce97a1b17d8a WatchSource:0}: Error finding container c5d43d150263b621d28465ef884b042972cefa6eea75fa4c8eb8ce97a1b17d8a: Status 404 returned error can't find the container with id c5d43d150263b621d28465ef884b042972cefa6eea75fa4c8eb8ce97a1b17d8a Sep 16 18:49:41.301956 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:41.301923 2563 generic.go:356] "Generic (PLEG): container finished" podID="92d7889a-6c53-4354-8c2e-76f4b64c20b3" containerID="45a7ab921632709ad64e7a688cd3f20d3eb3d1ab3333570a4ebfbf751eb44df2" exitCode=0 Sep 16 18:49:41.302138 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:41.302013 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92d7889a-6c53-4354-8c2e-76f4b64c20b3","Type":"ContainerDied","Data":"45a7ab921632709ad64e7a688cd3f20d3eb3d1ab3333570a4ebfbf751eb44df2"} Sep 16 18:49:41.302138 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:41.302096 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92d7889a-6c53-4354-8c2e-76f4b64c20b3","Type":"ContainerStarted","Data":"c5d43d150263b621d28465ef884b042972cefa6eea75fa4c8eb8ce97a1b17d8a"} Sep 16 18:49:42.310287 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:42.310248 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92d7889a-6c53-4354-8c2e-76f4b64c20b3","Type":"ContainerStarted","Data":"49fdc66d914629213ff7803751b6402463637921805a11543177ab4e76b515e9"} Sep 16 18:49:42.310658 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:42.310369 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92d7889a-6c53-4354-8c2e-76f4b64c20b3","Type":"ContainerStarted","Data":"13373a3e9cb807d1c86d04d4081ae2478724e3334bd3f07e940d5b731d741867"} Sep 16 18:49:42.310658 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:42.310413 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92d7889a-6c53-4354-8c2e-76f4b64c20b3","Type":"ContainerStarted","Data":"a46987c0deee80d641d2f1ac9d48b08466a5c3ffb6c4f9e11f0486a8feb1d93d"} Sep 16 18:49:42.310658 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:42.310449 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92d7889a-6c53-4354-8c2e-76f4b64c20b3","Type":"ContainerStarted","Data":"c935f04f422a7b37d197c041328e0fcfa27adb6872bf918406b030c67ebedcd2"} Sep 16 18:49:42.310658 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:42.310486 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92d7889a-6c53-4354-8c2e-76f4b64c20b3","Type":"ContainerStarted","Data":"7fad0fba6c4e8b3084134c4cd52d2485477be65a24f4086b5fcf1b46466d6f2f"} Sep 16 18:49:42.310658 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:42.310522 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"92d7889a-6c53-4354-8c2e-76f4b64c20b3","Type":"ContainerStarted","Data":"995dce2fa103bdb4c8d35b1af23552205a7628bcbc615321bd145ba6a5d3e13d"} Sep 16 18:49:42.340458 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:42.340401 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.321719283 podStartE2EDuration="2.340383341s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.664058ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:42.338618536 +0000 UTC m=+242.662311385" watchObservedRunningTime="2026-09-16 18:49:42.340383341 +0000 UTC m=+242.664076204" Sep 16 18:49:45.685403 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:45.685364 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:49:51.924357 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:51.924296 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:49:51.926670 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:51.926642 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/650deff9-b845-4ef3-acad-6bdee9667025-metrics-certs\") pod \"network-metrics-daemon-lq68v\" (UID: \"650deff9-b845-4ef3-acad-6bdee9667025\") " pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:49:52.034072 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:52.034036 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-f7n2q\"" Sep 16 18:49:52.041849 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:52.041815 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lq68v" Sep 16 18:49:52.313052 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:52.313021 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lq68v"] Sep 16 18:49:52.316569 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:49:52.316542 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod650deff9_b845_4ef3_acad_6bdee9667025.slice/crio-db6352b4e99c97bb288cd105c18703970e63c02d78cc1d81127da259a73f0221 WatchSource:0}: Error finding container db6352b4e99c97bb288cd105c18703970e63c02d78cc1d81127da259a73f0221: Status 404 returned error can't find the container with id db6352b4e99c97bb288cd105c18703970e63c02d78cc1d81127da259a73f0221 Sep 16 18:49:52.349603 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:52.349573 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lq68v" event={"ID":"650deff9-b845-4ef3-acad-6bdee9667025","Type":"ContainerStarted","Data":"db6352b4e99c97bb288cd105c18703970e63c02d78cc1d81127da259a73f0221"} Sep 16 18:49:54.360185 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:54.360146 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lq68v" event={"ID":"650deff9-b845-4ef3-acad-6bdee9667025","Type":"ContainerStarted","Data":"4c8a7808cf81b98156981747969b620c1e7b30539c6fd0638f6d066cfea2a671"} Sep 16 18:49:54.360673 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:54.360210 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lq68v" event={"ID":"650deff9-b845-4ef3-acad-6bdee9667025","Type":"ContainerStarted","Data":"0baddcf30e56088ea95c105872a495d52a87608f25baff48fb685ce8b9003194"} Sep 16 18:49:54.376730 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:49:54.376672 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-lq68v" podStartSLOduration=253.403203832 podStartE2EDuration="4m14.376655968s" totalImagesPullingTime="973.452136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:49:54.375802815 +0000 UTC m=+254.699495668" watchObservedRunningTime="2026-09-16 18:49:54.376655968 +0000 UTC m=+254.700348816" Sep 16 18:50:40.051502 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:50:40.051472 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 18:50:40.055281 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:50:40.055251 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 18:50:40.070358 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:50:40.070328 2563 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 18:50:40.685668 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:50:40.685620 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:50:40.702044 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:50:40.702015 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:50:41.556573 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:50:41.556543 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 18:55:33.679487 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.679435 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-hmw4n"] Sep 16 18:55:33.683053 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.683026 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hmw4n" Sep 16 18:55:33.685407 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.685353 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Sep 16 18:55:33.685600 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.685583 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-hqw4s\"" Sep 16 18:55:33.685600 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.685595 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 16 18:55:33.685911 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.685896 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Sep 16 18:55:33.685962 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.685928 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Sep 16 18:55:33.691505 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.691476 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hmw4n"] Sep 16 18:55:33.838275 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.838230 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b01050a3-2637-4a00-988c-fd0248a72ec3-model-cache\") pod \"s3-init-hmw4n\" (UID: \"b01050a3-2637-4a00-988c-fd0248a72ec3\") " pod="kserve/s3-init-hmw4n" Sep 16 18:55:33.838499 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.838378 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjl6d\" (UniqueName: \"kubernetes.io/projected/b01050a3-2637-4a00-988c-fd0248a72ec3-kube-api-access-rjl6d\") pod \"s3-init-hmw4n\" (UID: \"b01050a3-2637-4a00-988c-fd0248a72ec3\") " pod="kserve/s3-init-hmw4n" Sep 16 18:55:33.939967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.939877 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjl6d\" (UniqueName: \"kubernetes.io/projected/b01050a3-2637-4a00-988c-fd0248a72ec3-kube-api-access-rjl6d\") pod \"s3-init-hmw4n\" (UID: \"b01050a3-2637-4a00-988c-fd0248a72ec3\") " pod="kserve/s3-init-hmw4n" Sep 16 18:55:33.939967 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.939926 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b01050a3-2637-4a00-988c-fd0248a72ec3-model-cache\") pod \"s3-init-hmw4n\" (UID: \"b01050a3-2637-4a00-988c-fd0248a72ec3\") " pod="kserve/s3-init-hmw4n" Sep 16 18:55:33.940270 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.940254 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b01050a3-2637-4a00-988c-fd0248a72ec3-model-cache\") pod \"s3-init-hmw4n\" (UID: \"b01050a3-2637-4a00-988c-fd0248a72ec3\") " pod="kserve/s3-init-hmw4n" Sep 16 18:55:33.950532 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:33.950501 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjl6d\" (UniqueName: \"kubernetes.io/projected/b01050a3-2637-4a00-988c-fd0248a72ec3-kube-api-access-rjl6d\") pod \"s3-init-hmw4n\" (UID: \"b01050a3-2637-4a00-988c-fd0248a72ec3\") " pod="kserve/s3-init-hmw4n" Sep 16 18:55:34.004220 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:34.004179 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hmw4n" Sep 16 18:55:34.276206 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:34.276181 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hmw4n"] Sep 16 18:55:34.278663 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:55:34.278634 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb01050a3_2637_4a00_988c_fd0248a72ec3.slice/crio-013cf3d09e171037cfee1d2c734a68fdc25112526d008f0a9eac5bc4655b1f7c WatchSource:0}: Error finding container 013cf3d09e171037cfee1d2c734a68fdc25112526d008f0a9eac5bc4655b1f7c: Status 404 returned error can't find the container with id 013cf3d09e171037cfee1d2c734a68fdc25112526d008f0a9eac5bc4655b1f7c Sep 16 18:55:34.280628 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:34.280607 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 18:55:34.728827 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:34.728738 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hmw4n" event={"ID":"b01050a3-2637-4a00-988c-fd0248a72ec3","Type":"ContainerStarted","Data":"013cf3d09e171037cfee1d2c734a68fdc25112526d008f0a9eac5bc4655b1f7c"} Sep 16 18:55:37.744145 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:37.744059 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hmw4n" event={"ID":"b01050a3-2637-4a00-988c-fd0248a72ec3","Type":"ContainerStarted","Data":"6c86d34558cd494987bd815c8d0b87147952f4c5a657b315f7df39cdd049a575"} Sep 16 18:55:39.753564 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:39.753531 2563 generic.go:356] "Generic (PLEG): container finished" podID="b01050a3-2637-4a00-988c-fd0248a72ec3" containerID="6c86d34558cd494987bd815c8d0b87147952f4c5a657b315f7df39cdd049a575" exitCode=0 Sep 16 18:55:39.753986 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:39.753612 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hmw4n" event={"ID":"b01050a3-2637-4a00-988c-fd0248a72ec3","Type":"ContainerDied","Data":"6c86d34558cd494987bd815c8d0b87147952f4c5a657b315f7df39cdd049a575"} Sep 16 18:55:40.105251 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:40.105216 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 18:55:40.106469 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:40.106423 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 18:55:44.056883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:44.056861 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Sep 16 18:55:44.778560 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:44.778524 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hmw4n" event={"ID":"b01050a3-2637-4a00-988c-fd0248a72ec3","Type":"ContainerStarted","Data":"7adb9a06f8fe22ddd7849da53d4d926f033489e66caadf0cbe76512073a9a125"} Sep 16 18:55:44.797396 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:55:44.797347 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-hmw4n" podStartSLOduration=2.815912428 podStartE2EDuration="11.797331443s" totalImagesPullingTime="7.439077568s" totalInitContainerRuntime="1.542341447s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:44.796052524 +0000 UTC m=+605.119745372" watchObservedRunningTime="2026-09-16 18:55:44.797331443 +0000 UTC m=+605.121024291" Sep 16 18:56:50.044389 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:50.044352 2563 generic.go:356] "Generic (PLEG): container finished" podID="b01050a3-2637-4a00-988c-fd0248a72ec3" containerID="7adb9a06f8fe22ddd7849da53d4d926f033489e66caadf0cbe76512073a9a125" exitCode=0 Sep 16 18:56:50.044821 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:50.044414 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hmw4n" event={"ID":"b01050a3-2637-4a00-988c-fd0248a72ec3","Type":"ContainerDied","Data":"7adb9a06f8fe22ddd7849da53d4d926f033489e66caadf0cbe76512073a9a125"} Sep 16 18:56:51.176036 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:51.176011 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hmw4n" Sep 16 18:56:51.296516 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:51.296411 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b01050a3-2637-4a00-988c-fd0248a72ec3-model-cache\") pod \"b01050a3-2637-4a00-988c-fd0248a72ec3\" (UID: \"b01050a3-2637-4a00-988c-fd0248a72ec3\") " Sep 16 18:56:51.296696 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:51.296561 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjl6d\" (UniqueName: \"kubernetes.io/projected/b01050a3-2637-4a00-988c-fd0248a72ec3-kube-api-access-rjl6d\") pod \"b01050a3-2637-4a00-988c-fd0248a72ec3\" (UID: \"b01050a3-2637-4a00-988c-fd0248a72ec3\") " Sep 16 18:56:51.298849 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:51.298762 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b01050a3-2637-4a00-988c-fd0248a72ec3-kube-api-access-rjl6d" (OuterVolumeSpecName: "kube-api-access-rjl6d") pod "b01050a3-2637-4a00-988c-fd0248a72ec3" (UID: "b01050a3-2637-4a00-988c-fd0248a72ec3"). InnerVolumeSpecName "kube-api-access-rjl6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:56:51.315398 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:51.315355 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b01050a3-2637-4a00-988c-fd0248a72ec3-model-cache" (OuterVolumeSpecName: "model-cache") pod "b01050a3-2637-4a00-988c-fd0248a72ec3" (UID: "b01050a3-2637-4a00-988c-fd0248a72ec3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:56:51.397323 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:51.397288 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b01050a3-2637-4a00-988c-fd0248a72ec3-model-cache\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:56:51.397323 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:51.397319 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjl6d\" (UniqueName: \"kubernetes.io/projected/b01050a3-2637-4a00-988c-fd0248a72ec3-kube-api-access-rjl6d\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:56:52.053904 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:52.053866 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hmw4n" event={"ID":"b01050a3-2637-4a00-988c-fd0248a72ec3","Type":"ContainerDied","Data":"013cf3d09e171037cfee1d2c734a68fdc25112526d008f0a9eac5bc4655b1f7c"} Sep 16 18:56:52.053904 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:52.053890 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hmw4n" Sep 16 18:56:52.053904 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:56:52.053913 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="013cf3d09e171037cfee1d2c734a68fdc25112526d008f0a9eac5bc4655b1f7c" Sep 16 18:57:04.799462 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.799420 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6"] Sep 16 18:57:04.800070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.799750 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01050a3-2637-4a00-988c-fd0248a72ec3" containerName="s3-init" Sep 16 18:57:04.800070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.799761 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01050a3-2637-4a00-988c-fd0248a72ec3" containerName="s3-init" Sep 16 18:57:04.800070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.799770 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b01050a3-2637-4a00-988c-fd0248a72ec3" containerName="download-hf-model" Sep 16 18:57:04.800070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.799776 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b01050a3-2637-4a00-988c-fd0248a72ec3" containerName="download-hf-model" Sep 16 18:57:04.800070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.799843 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b01050a3-2637-4a00-988c-fd0248a72ec3" containerName="s3-init" Sep 16 18:57:04.802963 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.802939 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:04.805388 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.805352 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\"" Sep 16 18:57:04.805522 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.805475 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\"" Sep 16 18:57:04.805696 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.805678 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-fxpzh\"" Sep 16 18:57:04.805760 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.805708 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Sep 16 18:57:04.806001 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.805983 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\"" Sep 16 18:57:04.806099 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.805984 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\"" Sep 16 18:57:04.806190 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.806176 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Sep 16 18:57:04.814572 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.814549 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6"] Sep 16 18:57:04.910219 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.910180 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:04.910426 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.910234 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:04.910426 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.910287 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h87fh\" (UniqueName: \"kubernetes.io/projected/b1451641-e30a-461d-a14f-fa82d5243955-kube-api-access-h87fh\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:04.910426 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.910310 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1451641-e30a-461d-a14f-fa82d5243955-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:04.910426 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:04.910363 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1451641-e30a-461d-a14f-fa82d5243955-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.011650 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.011608 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.011871 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.011667 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.011871 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.011705 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h87fh\" (UniqueName: \"kubernetes.io/projected/b1451641-e30a-461d-a14f-fa82d5243955-kube-api-access-h87fh\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.011871 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.011733 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1451641-e30a-461d-a14f-fa82d5243955-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.011871 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.011772 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1451641-e30a-461d-a14f-fa82d5243955-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.012093 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:57:05.011966 2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found Sep 16 18:57:05.012093 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:57:05.012044 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1451641-e30a-461d-a14f-fa82d5243955-proxy-tls podName:b1451641-e30a-461d-a14f-fa82d5243955 nodeName:}" failed. No retries permitted until 2026-09-16 18:57:05.512019895 +0000 UTC m=+685.835712734 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b1451641-e30a-461d-a14f-fa82d5243955-proxy-tls") pod "isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" (UID: "b1451641-e30a-461d-a14f-fa82d5243955") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found Sep 16 18:57:05.012464 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.012435 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.012883 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.012825 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1451641-e30a-461d-a14f-fa82d5243955-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.013014 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.012864 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.024462 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.024423 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h87fh\" (UniqueName: \"kubernetes.io/projected/b1451641-e30a-461d-a14f-fa82d5243955-kube-api-access-h87fh\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.272621 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.272579 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h"] Sep 16 18:57:05.276673 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.276649 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.279741 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.279719 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\"" Sep 16 18:57:05.280087 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.280061 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\"" Sep 16 18:57:05.286742 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.286709 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h"] Sep 16 18:57:05.379021 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.378980 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r"] Sep 16 18:57:05.383177 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.383150 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.385543 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.385518 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\"" Sep 16 18:57:05.385675 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.385639 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\"" Sep 16 18:57:05.397941 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.397913 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r"] Sep 16 18:57:05.415212 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.415177 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.415610 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.415582 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.415713 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.415633 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjmsp\" (UniqueName: \"kubernetes.io/projected/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-api-access-sjmsp\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.415713 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.415668 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.415713 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.415694 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.516999 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.516944 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.516999 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517006 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq2zp\" (UniqueName: \"kubernetes.io/projected/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-api-access-pq2zp\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.517296 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517097 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.517296 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517170 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.517296 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517202 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.517296 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517239 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.517489 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517300 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjmsp\" (UniqueName: \"kubernetes.io/projected/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-api-access-sjmsp\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.517489 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517344 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1451641-e30a-461d-a14f-fa82d5243955-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.517489 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.517489 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517397 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.517705 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517616 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.517765 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517744 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.518020 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.517992 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.518153 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.518023 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.519954 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.519930 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.520172 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.520149 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1451641-e30a-461d-a14f-fa82d5243955-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.526002 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.525943 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjmsp\" (UniqueName: \"kubernetes.io/projected/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-api-access-sjmsp\") pod \"isvc-xgboost-graph-predictor-555fc69d56-2k99h\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.589023 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.588980 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:05.618158 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.618116 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.618158 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.618158 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq2zp\" (UniqueName: \"kubernetes.io/projected/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-api-access-pq2zp\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.618382 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.618190 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.618382 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.618237 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.618382 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.618279 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.618382 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:57:05.618358 2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-serving-cert: secret "isvc-sklearn-graph-2-predictor-serving-cert" not found Sep 16 18:57:05.618584 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:57:05.618447 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-proxy-tls podName:d21f1d33-0f7f-4d9f-9d55-f04c961d3161 nodeName:}" failed. No retries permitted until 2026-09-16 18:57:06.118425633 +0000 UTC m=+686.442118464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-proxy-tls") pod "isvc-sklearn-graph-2-predictor-99489b495-hg58r" (UID: "d21f1d33-0f7f-4d9f-9d55-f04c961d3161") : secret "isvc-sklearn-graph-2-predictor-serving-cert" not found Sep 16 18:57:05.618647 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.618632 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.618935 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.618913 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.619128 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.619107 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.629658 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.629625 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq2zp\" (UniqueName: \"kubernetes.io/projected/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-api-access-pq2zp\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:05.714304 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.714272 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:05.869879 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.869776 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h"] Sep 16 18:57:05.873139 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:57:05.873110 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8ce26d9_bfc1_4555_bc6e_47b6fc0e234c.slice/crio-f102b2e0dbda2e48bd3c022aa6549aa6e91909d3187e95e409fea9a85bb3b62f WatchSource:0}: Error finding container f102b2e0dbda2e48bd3c022aa6549aa6e91909d3187e95e409fea9a85bb3b62f: Status 404 returned error can't find the container with id f102b2e0dbda2e48bd3c022aa6549aa6e91909d3187e95e409fea9a85bb3b62f Sep 16 18:57:05.991134 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:05.991107 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6"] Sep 16 18:57:05.993380 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:57:05.993341 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1451641_e30a_461d_a14f_fa82d5243955.slice/crio-6a904a42868dd993bcd3ba6dd6427a03ec4943a15f00247415048190aa095073 WatchSource:0}: Error finding container 6a904a42868dd993bcd3ba6dd6427a03ec4943a15f00247415048190aa095073: Status 404 returned error can't find the container with id 6a904a42868dd993bcd3ba6dd6427a03ec4943a15f00247415048190aa095073 Sep 16 18:57:06.114010 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:06.113965 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" event={"ID":"b1451641-e30a-461d-a14f-fa82d5243955","Type":"ContainerStarted","Data":"4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e"} Sep 16 18:57:06.114222 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:06.114051 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" event={"ID":"b1451641-e30a-461d-a14f-fa82d5243955","Type":"ContainerStarted","Data":"6a904a42868dd993bcd3ba6dd6427a03ec4943a15f00247415048190aa095073"} Sep 16 18:57:06.115869 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:06.115816 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" event={"ID":"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c","Type":"ContainerStarted","Data":"afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12"} Sep 16 18:57:06.115984 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:06.115891 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" event={"ID":"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c","Type":"ContainerStarted","Data":"f102b2e0dbda2e48bd3c022aa6549aa6e91909d3187e95e409fea9a85bb3b62f"} Sep 16 18:57:06.123173 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:06.123120 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:06.125907 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:06.125883 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-99489b495-hg58r\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:06.297805 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:06.297766 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:06.593125 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:06.593090 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r"] Sep 16 18:57:06.596276 ip-10-0-134-125 kubenswrapper[2563]: W0916 18:57:06.596246 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21f1d33_0f7f_4d9f_9d55_f04c961d3161.slice/crio-8ef3ab27fa7947ccb7ecf56b908312a4c01f34be47e3856fcc22ae3bbd6f4cb0 WatchSource:0}: Error finding container 8ef3ab27fa7947ccb7ecf56b908312a4c01f34be47e3856fcc22ae3bbd6f4cb0: Status 404 returned error can't find the container with id 8ef3ab27fa7947ccb7ecf56b908312a4c01f34be47e3856fcc22ae3bbd6f4cb0 Sep 16 18:57:07.122314 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:07.122276 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" event={"ID":"d21f1d33-0f7f-4d9f-9d55-f04c961d3161","Type":"ContainerStarted","Data":"fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3"} Sep 16 18:57:07.122788 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:07.122344 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" event={"ID":"d21f1d33-0f7f-4d9f-9d55-f04c961d3161","Type":"ContainerStarted","Data":"8ef3ab27fa7947ccb7ecf56b908312a4c01f34be47e3856fcc22ae3bbd6f4cb0"} Sep 16 18:57:10.143772 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:10.143742 2563 generic.go:356] "Generic (PLEG): container finished" podID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerID="afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12" exitCode=0 Sep 16 18:57:10.144143 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:10.143811 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" event={"ID":"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c","Type":"ContainerDied","Data":"afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12"} Sep 16 18:57:10.145686 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:10.145664 2563 generic.go:356] "Generic (PLEG): container finished" podID="b1451641-e30a-461d-a14f-fa82d5243955" containerID="4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e" exitCode=0 Sep 16 18:57:10.145818 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:10.145699 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" event={"ID":"b1451641-e30a-461d-a14f-fa82d5243955","Type":"ContainerDied","Data":"4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e"} Sep 16 18:57:11.158277 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:11.158238 2563 generic.go:356] "Generic (PLEG): container finished" podID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerID="fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3" exitCode=0 Sep 16 18:57:11.158758 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:11.158368 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" event={"ID":"d21f1d33-0f7f-4d9f-9d55-f04c961d3161","Type":"ContainerDied","Data":"fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3"} Sep 16 18:57:38.329701 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:38.329669 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" event={"ID":"b1451641-e30a-461d-a14f-fa82d5243955","Type":"ContainerStarted","Data":"7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d"} Sep 16 18:57:39.352573 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:39.352433 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" event={"ID":"d21f1d33-0f7f-4d9f-9d55-f04c961d3161","Type":"ContainerStarted","Data":"8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693"} Sep 16 18:57:39.356505 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:39.356402 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" event={"ID":"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c","Type":"ContainerStarted","Data":"6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43"} Sep 16 18:57:41.370825 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.370688 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" event={"ID":"b1451641-e30a-461d-a14f-fa82d5243955","Type":"ContainerStarted","Data":"e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913"} Sep 16 18:57:41.370825 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.370774 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:41.370825 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.370812 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:41.372303 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.372262 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Sep 16 18:57:41.373181 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.373158 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" event={"ID":"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c","Type":"ContainerStarted","Data":"3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc"} Sep 16 18:57:41.373689 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.373671 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:41.373872 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.373860 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:41.375040 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.375010 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Sep 16 18:57:41.376327 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.376308 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" event={"ID":"d21f1d33-0f7f-4d9f-9d55-f04c961d3161","Type":"ContainerStarted","Data":"77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a"} Sep 16 18:57:41.376460 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.376442 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:41.400234 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.400175 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podStartSLOduration=2.546698846 podStartE2EDuration="37.400157446s" totalImagesPullingTime="30.796690869s" totalInitContainerRuntime="4.056767731s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:41.398151002 +0000 UTC m=+721.721843853" watchObservedRunningTime="2026-09-16 18:57:41.400157446 +0000 UTC m=+721.723850295" Sep 16 18:57:41.420676 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.420602 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podStartSLOduration=2.470837093 podStartE2EDuration="36.420580322s" totalImagesPullingTime="29.948504763s" totalInitContainerRuntime="4.001238466s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:41.419533598 +0000 UTC m=+721.743226461" watchObservedRunningTime="2026-09-16 18:57:41.420580322 +0000 UTC m=+721.744273173" Sep 16 18:57:41.442141 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:41.442053 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podStartSLOduration=1.616489335 podStartE2EDuration="36.442030602s" totalImagesPullingTime="30.980303s" totalInitContainerRuntime="3.845238267s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:57:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:57:41.441508582 +0000 UTC m=+721.765201431" watchObservedRunningTime="2026-09-16 18:57:41.442030602 +0000 UTC m=+721.765723455" Sep 16 18:57:42.380655 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:42.380609 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Sep 16 18:57:42.381167 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:42.380697 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Sep 16 18:57:42.381167 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:42.380886 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:42.382012 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:42.381987 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Sep 16 18:57:43.385070 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:43.385023 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Sep 16 18:57:43.385595 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:43.385022 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Sep 16 18:57:47.386066 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:47.386032 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:57:47.386536 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:47.386510 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Sep 16 18:57:48.389613 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:48.389575 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:57:48.390677 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:48.390640 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Sep 16 18:57:48.390873 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:48.390853 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:57:48.391358 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:48.391339 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Sep 16 18:57:57.387556 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:57.387510 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Sep 16 18:57:58.390740 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:58.390700 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Sep 16 18:57:58.391932 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:57:58.391906 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Sep 16 18:58:07.387264 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:07.387220 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Sep 16 18:58:08.391423 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:08.391384 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Sep 16 18:58:08.391423 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:08.391409 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Sep 16 18:58:17.386522 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:17.386478 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Sep 16 18:58:18.391334 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:18.391294 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Sep 16 18:58:18.391696 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:18.391298 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Sep 16 18:58:27.386992 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:27.386948 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Sep 16 18:58:28.391239 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:28.391197 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Sep 16 18:58:28.391674 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:28.391358 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Sep 16 18:58:37.386946 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:37.386898 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Sep 16 18:58:38.391708 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:38.391666 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Sep 16 18:58:38.392240 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:38.391666 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Sep 16 18:58:47.386871 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:47.386815 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Sep 16 18:58:48.391410 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:48.391365 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Sep 16 18:58:48.391848 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:48.391812 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:58:57.387306 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:57.387273 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:58:58.391896 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:58:58.391866 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:59:34.959439 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:34.959397 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r"] Sep 16 18:59:34.960046 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:34.959762 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" containerID="cri-o://8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693" gracePeriod=30 Sep 16 18:59:34.960046 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:34.959781 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kube-rbac-proxy" containerID="cri-o://77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a" gracePeriod=30 Sep 16 18:59:35.006658 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.006625 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6"] Sep 16 18:59:35.006921 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.006884 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" containerID="cri-o://7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d" gracePeriod=30 Sep 16 18:59:35.007060 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.006920 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kube-rbac-proxy" containerID="cri-o://e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913" gracePeriod=30 Sep 16 18:59:35.033710 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.033676 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h"] Sep 16 18:59:35.034010 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.033977 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" containerID="cri-o://6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43" gracePeriod=30 Sep 16 18:59:35.034010 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.033989 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kube-rbac-proxy" containerID="cri-o://3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc" gracePeriod=30 Sep 16 18:59:35.878186 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.878151 2563 generic.go:356] "Generic (PLEG): container finished" podID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerID="77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a" exitCode=2 Sep 16 18:59:35.878368 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.878228 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" event={"ID":"d21f1d33-0f7f-4d9f-9d55-f04c961d3161","Type":"ContainerDied","Data":"77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a"} Sep 16 18:59:35.880876 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.880849 2563 generic.go:356] "Generic (PLEG): container finished" podID="b1451641-e30a-461d-a14f-fa82d5243955" containerID="e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913" exitCode=2 Sep 16 18:59:35.880876 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.880866 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" event={"ID":"b1451641-e30a-461d-a14f-fa82d5243955","Type":"ContainerDied","Data":"e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913"} Sep 16 18:59:35.882969 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.882947 2563 generic.go:356] "Generic (PLEG): container finished" podID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerID="3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc" exitCode=2 Sep 16 18:59:35.883072 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:35.882997 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" event={"ID":"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c","Type":"ContainerDied","Data":"3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc"} Sep 16 18:59:37.381509 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:37.381460 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused" Sep 16 18:59:37.386891 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:37.386862 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Sep 16 18:59:38.385992 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:38.385947 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" Sep 16 18:59:38.386396 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:38.385962 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused" Sep 16 18:59:38.391161 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:38.391125 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Sep 16 18:59:38.391322 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:38.391300 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Sep 16 18:59:39.185485 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.185452 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:59:39.321157 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.321123 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-rbac-proxy-sar-config\") pod \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " Sep 16 18:59:39.321347 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.321176 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-cabundle-cert\") pod \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " Sep 16 18:59:39.321347 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.321235 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sjmsp\" (UniqueName: \"kubernetes.io/projected/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-api-access-sjmsp\") pod \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " Sep 16 18:59:39.321465 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.321440 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kserve-provision-location\") pod \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " Sep 16 18:59:39.321511 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.321491 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-proxy-tls\") pod \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\" (UID: \"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c\") " Sep 16 18:59:39.321620 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.321592 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" (UID: "b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:39.321620 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.321610 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" (UID: "b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:39.321773 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.321753 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" (UID: "b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:59:39.321946 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.321928 2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:39.322016 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.321971 2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-cabundle-cert\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:39.322016 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.321983 2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kserve-provision-location\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:39.323524 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.323499 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-api-access-sjmsp" (OuterVolumeSpecName: "kube-api-access-sjmsp") pod "b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" (UID: "b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c"). InnerVolumeSpecName "kube-api-access-sjmsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:59:39.323633 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.323581 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" (UID: "b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:59:39.422437 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.422396 2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-proxy-tls\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:39.422437 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.422431 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjmsp\" (UniqueName: \"kubernetes.io/projected/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c-kube-api-access-sjmsp\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:39.902915 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.902827 2563 generic.go:356] "Generic (PLEG): container finished" podID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerID="6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43" exitCode=0 Sep 16 18:59:39.902915 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.902908 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" Sep 16 18:59:39.903131 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.902937 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" event={"ID":"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c","Type":"ContainerDied","Data":"6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43"} Sep 16 18:59:39.903131 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.903002 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h" event={"ID":"b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c","Type":"ContainerDied","Data":"f102b2e0dbda2e48bd3c022aa6549aa6e91909d3187e95e409fea9a85bb3b62f"} Sep 16 18:59:39.903131 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.903031 2563 scope.go:117] "RemoveContainer" containerID="3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc" Sep 16 18:59:39.912322 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.912299 2563 scope.go:117] "RemoveContainer" containerID="6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43" Sep 16 18:59:39.920523 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.920501 2563 scope.go:117] "RemoveContainer" containerID="afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12" Sep 16 18:59:39.924259 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.924233 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h"] Sep 16 18:59:39.928627 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.928601 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-2k99h"] Sep 16 18:59:39.929386 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.929368 2563 scope.go:117] "RemoveContainer" containerID="3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc" Sep 16 18:59:39.929732 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:59:39.929714 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc\": container with ID starting with 3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc not found: ID does not exist" containerID="3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc" Sep 16 18:59:39.929787 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.929742 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc"} err="failed to get container status \"3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc\": rpc error: code = NotFound desc = could not find container \"3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc\": container with ID starting with 3bed8edcd208a79fba09d8d96587b94d790961c409dea7952445fd52ab18babc not found: ID does not exist" Sep 16 18:59:39.929787 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.929762 2563 scope.go:117] "RemoveContainer" containerID="6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43" Sep 16 18:59:39.930169 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:59:39.930141 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43\": container with ID starting with 6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43 not found: ID does not exist" containerID="6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43" Sep 16 18:59:39.930260 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.930174 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43"} err="failed to get container status \"6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43\": rpc error: code = NotFound desc = could not find container \"6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43\": container with ID starting with 6ccf5094b9f50a75d4bdc03a165778131d2ec8245d48f8de40f6156a1f762b43 not found: ID does not exist" Sep 16 18:59:39.930260 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.930193 2563 scope.go:117] "RemoveContainer" containerID="afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12" Sep 16 18:59:39.930481 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:59:39.930463 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12\": container with ID starting with afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12 not found: ID does not exist" containerID="afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12" Sep 16 18:59:39.930519 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:39.930486 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12"} err="failed to get container status \"afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12\": rpc error: code = NotFound desc = could not find container \"afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12\": container with ID starting with afe08978ff3e9828f1a75ec6534769578eaa87f40efcada4986b0ef3e830ad12 not found: ID does not exist" Sep 16 18:59:40.336569 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.336536 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" path="/var/lib/kubelet/pods/b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c/volumes" Sep 16 18:59:40.643843 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.643814 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:59:40.673993 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.673970 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:59:40.834398 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834356 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1451641-e30a-461d-a14f-fa82d5243955-proxy-tls\") pod \"b1451641-e30a-461d-a14f-fa82d5243955\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " Sep 16 18:59:40.834398 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834398 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-cabundle-cert\") pod \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " Sep 16 18:59:40.834619 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834440 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-rbac-proxy-sar-config\") pod \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " Sep 16 18:59:40.834619 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834470 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-cabundle-cert\") pod \"b1451641-e30a-461d-a14f-fa82d5243955\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " Sep 16 18:59:40.834619 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834514 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1451641-e30a-461d-a14f-fa82d5243955-kserve-provision-location\") pod \"b1451641-e30a-461d-a14f-fa82d5243955\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " Sep 16 18:59:40.834619 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834550 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pq2zp\" (UniqueName: \"kubernetes.io/projected/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-api-access-pq2zp\") pod \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " Sep 16 18:59:40.834619 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834594 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h87fh\" (UniqueName: \"kubernetes.io/projected/b1451641-e30a-461d-a14f-fa82d5243955-kube-api-access-h87fh\") pod \"b1451641-e30a-461d-a14f-fa82d5243955\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " Sep 16 18:59:40.834914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834643 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-proxy-tls\") pod \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " Sep 16 18:59:40.834914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834674 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-kube-rbac-proxy-sar-config\") pod \"b1451641-e30a-461d-a14f-fa82d5243955\" (UID: \"b1451641-e30a-461d-a14f-fa82d5243955\") " Sep 16 18:59:40.834914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834711 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kserve-provision-location\") pod \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\" (UID: \"d21f1d33-0f7f-4d9f-9d55-f04c961d3161\") " Sep 16 18:59:40.834914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834865 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1451641-e30a-461d-a14f-fa82d5243955-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1451641-e30a-461d-a14f-fa82d5243955" (UID: "b1451641-e30a-461d-a14f-fa82d5243955"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:59:40.834914 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834875 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d21f1d33-0f7f-4d9f-9d55-f04c961d3161" (UID: "d21f1d33-0f7f-4d9f-9d55-f04c961d3161"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:40.835181 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834918 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1451641-e30a-461d-a14f-fa82d5243955" (UID: "b1451641-e30a-461d-a14f-fa82d5243955"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:40.835181 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.834947 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d21f1d33-0f7f-4d9f-9d55-f04c961d3161" (UID: "d21f1d33-0f7f-4d9f-9d55-f04c961d3161"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:40.835288 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.835195 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d21f1d33-0f7f-4d9f-9d55-f04c961d3161" (UID: "d21f1d33-0f7f-4d9f-9d55-f04c961d3161"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 18:59:40.835288 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.835241 2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-cabundle-cert\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:40.835288 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.835246 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1451641-e30a-461d-a14f-fa82d5243955" (UID: "b1451641-e30a-461d-a14f-fa82d5243955"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 18:59:40.835288 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.835263 2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:40.835288 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.835278 2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-cabundle-cert\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:40.835545 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.835292 2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1451641-e30a-461d-a14f-fa82d5243955-kserve-provision-location\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:40.836720 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.836688 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1451641-e30a-461d-a14f-fa82d5243955-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1451641-e30a-461d-a14f-fa82d5243955" (UID: "b1451641-e30a-461d-a14f-fa82d5243955"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:59:40.836895 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.836870 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-api-access-pq2zp" (OuterVolumeSpecName: "kube-api-access-pq2zp") pod "d21f1d33-0f7f-4d9f-9d55-f04c961d3161" (UID: "d21f1d33-0f7f-4d9f-9d55-f04c961d3161"). InnerVolumeSpecName "kube-api-access-pq2zp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:59:40.836979 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.836964 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d21f1d33-0f7f-4d9f-9d55-f04c961d3161" (UID: "d21f1d33-0f7f-4d9f-9d55-f04c961d3161"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 18:59:40.837021 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.837006 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1451641-e30a-461d-a14f-fa82d5243955-kube-api-access-h87fh" (OuterVolumeSpecName: "kube-api-access-h87fh") pod "b1451641-e30a-461d-a14f-fa82d5243955" (UID: "b1451641-e30a-461d-a14f-fa82d5243955"). InnerVolumeSpecName "kube-api-access-h87fh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 18:59:40.909282 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.909195 2563 generic.go:356] "Generic (PLEG): container finished" podID="b1451641-e30a-461d-a14f-fa82d5243955" containerID="7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d" exitCode=0 Sep 16 18:59:40.909282 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.909258 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" Sep 16 18:59:40.909515 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.909277 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" event={"ID":"b1451641-e30a-461d-a14f-fa82d5243955","Type":"ContainerDied","Data":"7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d"} Sep 16 18:59:40.909515 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.909340 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6" event={"ID":"b1451641-e30a-461d-a14f-fa82d5243955","Type":"ContainerDied","Data":"6a904a42868dd993bcd3ba6dd6427a03ec4943a15f00247415048190aa095073"} Sep 16 18:59:40.909515 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.909375 2563 scope.go:117] "RemoveContainer" containerID="e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913" Sep 16 18:59:40.911544 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.911519 2563 generic.go:356] "Generic (PLEG): container finished" podID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerID="8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693" exitCode=0 Sep 16 18:59:40.911641 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.911567 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" event={"ID":"d21f1d33-0f7f-4d9f-9d55-f04c961d3161","Type":"ContainerDied","Data":"8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693"} Sep 16 18:59:40.911641 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.911622 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" event={"ID":"d21f1d33-0f7f-4d9f-9d55-f04c961d3161","Type":"ContainerDied","Data":"8ef3ab27fa7947ccb7ecf56b908312a4c01f34be47e3856fcc22ae3bbd6f4cb0"} Sep 16 18:59:40.911641 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.911630 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r" Sep 16 18:59:40.922017 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.920472 2563 scope.go:117] "RemoveContainer" containerID="7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d" Sep 16 18:59:40.929675 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.929656 2563 scope.go:117] "RemoveContainer" containerID="4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e" Sep 16 18:59:40.935625 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.935606 2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-proxy-tls\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:40.935625 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.935627 2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1451641-e30a-461d-a14f-fa82d5243955-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:40.935738 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.935637 2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kserve-provision-location\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:40.935738 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.935646 2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1451641-e30a-461d-a14f-fa82d5243955-proxy-tls\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:40.935738 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.935655 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq2zp\" (UniqueName: \"kubernetes.io/projected/d21f1d33-0f7f-4d9f-9d55-f04c961d3161-kube-api-access-pq2zp\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:40.935738 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.935663 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h87fh\" (UniqueName: \"kubernetes.io/projected/b1451641-e30a-461d-a14f-fa82d5243955-kube-api-access-h87fh\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 18:59:40.936340 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.936321 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6"] Sep 16 18:59:40.937975 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.937959 2563 scope.go:117] "RemoveContainer" containerID="e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913" Sep 16 18:59:40.938251 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:59:40.938230 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913\": container with ID starting with e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913 not found: ID does not exist" containerID="e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913" Sep 16 18:59:40.938307 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.938258 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913"} err="failed to get container status \"e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913\": rpc error: code = NotFound desc = could not find container \"e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913\": container with ID starting with e41a2941922c9a3c8f398751b1ec563a7b8e1a739b325812c285f1a1aa641913 not found: ID does not exist" Sep 16 18:59:40.938307 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.938274 2563 scope.go:117] "RemoveContainer" containerID="7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d" Sep 16 18:59:40.938531 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:59:40.938514 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d\": container with ID starting with 7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d not found: ID does not exist" containerID="7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d" Sep 16 18:59:40.938574 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.938536 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d"} err="failed to get container status \"7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d\": rpc error: code = NotFound desc = could not find container \"7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d\": container with ID starting with 7674e80f0be20642d38968e371f45da63f803c4c38d30050e5dc343a6a1cb72d not found: ID does not exist" Sep 16 18:59:40.938574 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.938551 2563 scope.go:117] "RemoveContainer" containerID="4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e" Sep 16 18:59:40.938776 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:59:40.938760 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e\": container with ID starting with 4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e not found: ID does not exist" containerID="4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e" Sep 16 18:59:40.938820 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.938778 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e"} err="failed to get container status \"4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e\": rpc error: code = NotFound desc = could not find container \"4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e\": container with ID starting with 4c2774fc7852a62fbae824f9f91214e12dca26fdd419a2154f531734be95ae5e not found: ID does not exist" Sep 16 18:59:40.938820 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.938790 2563 scope.go:117] "RemoveContainer" containerID="77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a" Sep 16 18:59:40.941928 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.941905 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-597b9ffcf8-2zzm6"] Sep 16 18:59:40.946271 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.946242 2563 scope.go:117] "RemoveContainer" containerID="8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693" Sep 16 18:59:40.952659 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.952638 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r"] Sep 16 18:59:40.954889 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.954856 2563 scope.go:117] "RemoveContainer" containerID="fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3" Sep 16 18:59:40.956918 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.956898 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-99489b495-hg58r"] Sep 16 18:59:40.962627 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.962609 2563 scope.go:117] "RemoveContainer" containerID="77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a" Sep 16 18:59:40.962915 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:59:40.962891 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a\": container with ID starting with 77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a not found: ID does not exist" containerID="77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a" Sep 16 18:59:40.963016 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.962920 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a"} err="failed to get container status \"77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a\": rpc error: code = NotFound desc = could not find container \"77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a\": container with ID starting with 77050bf5db150b3fab91341d808668f79463ca9834982b3e30bcf1a661f71c4a not found: ID does not exist" Sep 16 18:59:40.963016 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.962938 2563 scope.go:117] "RemoveContainer" containerID="8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693" Sep 16 18:59:40.963194 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:59:40.963178 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693\": container with ID starting with 8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693 not found: ID does not exist" containerID="8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693" Sep 16 18:59:40.963239 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.963196 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693"} err="failed to get container status \"8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693\": rpc error: code = NotFound desc = could not find container \"8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693\": container with ID starting with 8ba37915a0aba0c3892841e75ec86a4ef1be52fc1f9aa84f915c9f8a97e1d693 not found: ID does not exist" Sep 16 18:59:40.963239 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.963208 2563 scope.go:117] "RemoveContainer" containerID="fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3" Sep 16 18:59:40.963423 ip-10-0-134-125 kubenswrapper[2563]: E0916 18:59:40.963409 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3\": container with ID starting with fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3 not found: ID does not exist" containerID="fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3" Sep 16 18:59:40.963467 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:40.963426 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3"} err="failed to get container status \"fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3\": rpc error: code = NotFound desc = could not find container \"fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3\": container with ID starting with fab6e33ee49e98ea873accdc93c5012a93a487c32c0f97e24eca07a4572601b3 not found: ID does not exist" Sep 16 18:59:42.338608 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:42.338571 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1451641-e30a-461d-a14f-fa82d5243955" path="/var/lib/kubelet/pods/b1451641-e30a-461d-a14f-fa82d5243955/volumes" Sep 16 18:59:42.339173 ip-10-0-134-125 kubenswrapper[2563]: I0916 18:59:42.339154 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" path="/var/lib/kubelet/pods/d21f1d33-0f7f-4d9f-9d55-f04c961d3161/volumes" Sep 16 19:00:40.147250 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:00:40.146914 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:00:40.148765 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:00:40.148739 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:05:40.175392 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:05:40.175355 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:05:40.177441 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:05:40.177418 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:10:40.206042 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:10:40.206012 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:10:40.208129 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:10:40.208103 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:15:40.238276 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:15:40.238246 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:15:40.238763 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:15:40.238370 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:20:40.265200 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:20:40.265171 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:20:40.267558 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:20:40.267221 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:25:40.293132 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:25:40.293016 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:25:40.296957 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:25:40.295273 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:30:40.322610 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:30:40.322501 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:30:40.326472 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:30:40.324926 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:35:40.354611 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:35:40.354500 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:35:40.358480 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:35:40.357486 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:36:35.892299 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892211 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dnf7m/must-gather-d7hch"] Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892524 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="storage-initializer" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892535 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="storage-initializer" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892543 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="storage-initializer" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892548 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="storage-initializer" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892556 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="storage-initializer" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892562 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="storage-initializer" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892568 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892573 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892580 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kube-rbac-proxy" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892585 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kube-rbac-proxy" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892592 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kube-rbac-proxy" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892597 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kube-rbac-proxy" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892604 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kube-rbac-proxy" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892609 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kube-rbac-proxy" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892622 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892627 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892632 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892637 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892688 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kserve-container" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892700 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kserve-container" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892708 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kserve-container" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892713 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8ce26d9-bfc1-4555-bc6e-47b6fc0e234c" containerName="kube-rbac-proxy" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892718 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d21f1d33-0f7f-4d9f-9d55-f04c961d3161" containerName="kube-rbac-proxy" Sep 16 19:36:35.892752 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.892724 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1451641-e30a-461d-a14f-fa82d5243955" containerName="kube-rbac-proxy" Sep 16 19:36:35.895981 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.895962 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnf7m/must-gather-d7hch" Sep 16 19:36:35.898131 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.898105 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dnf7m\"/\"default-dockercfg-nrwbr\"" Sep 16 19:36:35.898255 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.898166 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dnf7m\"/\"kube-root-ca.crt\"" Sep 16 19:36:35.898744 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.898725 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dnf7m\"/\"openshift-service-ca.crt\"" Sep 16 19:36:35.905614 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.905588 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnf7m/must-gather-d7hch"] Sep 16 19:36:35.985276 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.985235 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93f54cc4-7c15-4623-8b33-1b96d26ee470-must-gather-output\") pod \"must-gather-d7hch\" (UID: \"93f54cc4-7c15-4623-8b33-1b96d26ee470\") " pod="openshift-must-gather-dnf7m/must-gather-d7hch" Sep 16 19:36:35.985468 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:35.985329 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbq5r\" (UniqueName: \"kubernetes.io/projected/93f54cc4-7c15-4623-8b33-1b96d26ee470-kube-api-access-pbq5r\") pod \"must-gather-d7hch\" (UID: \"93f54cc4-7c15-4623-8b33-1b96d26ee470\") " pod="openshift-must-gather-dnf7m/must-gather-d7hch" Sep 16 19:36:36.086466 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:36.086425 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbq5r\" (UniqueName: \"kubernetes.io/projected/93f54cc4-7c15-4623-8b33-1b96d26ee470-kube-api-access-pbq5r\") pod \"must-gather-d7hch\" (UID: \"93f54cc4-7c15-4623-8b33-1b96d26ee470\") " pod="openshift-must-gather-dnf7m/must-gather-d7hch" Sep 16 19:36:36.086662 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:36.086482 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93f54cc4-7c15-4623-8b33-1b96d26ee470-must-gather-output\") pod \"must-gather-d7hch\" (UID: \"93f54cc4-7c15-4623-8b33-1b96d26ee470\") " pod="openshift-must-gather-dnf7m/must-gather-d7hch" Sep 16 19:36:36.086792 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:36.086777 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93f54cc4-7c15-4623-8b33-1b96d26ee470-must-gather-output\") pod \"must-gather-d7hch\" (UID: \"93f54cc4-7c15-4623-8b33-1b96d26ee470\") " pod="openshift-must-gather-dnf7m/must-gather-d7hch" Sep 16 19:36:36.095381 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:36.095354 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbq5r\" (UniqueName: \"kubernetes.io/projected/93f54cc4-7c15-4623-8b33-1b96d26ee470-kube-api-access-pbq5r\") pod \"must-gather-d7hch\" (UID: \"93f54cc4-7c15-4623-8b33-1b96d26ee470\") " pod="openshift-must-gather-dnf7m/must-gather-d7hch" Sep 16 19:36:36.215514 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:36.215415 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnf7m/must-gather-d7hch" Sep 16 19:36:36.487222 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:36.487194 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dnf7m/must-gather-d7hch"] Sep 16 19:36:36.489776 ip-10-0-134-125 kubenswrapper[2563]: W0916 19:36:36.489748 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93f54cc4_7c15_4623_8b33_1b96d26ee470.slice/crio-4def1462e4f30acc5eb324bd3ceca7376443bf2625b3bf4807d11f6dbf1f4c67 WatchSource:0}: Error finding container 4def1462e4f30acc5eb324bd3ceca7376443bf2625b3bf4807d11f6dbf1f4c67: Status 404 returned error can't find the container with id 4def1462e4f30acc5eb324bd3ceca7376443bf2625b3bf4807d11f6dbf1f4c67 Sep 16 19:36:36.491818 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:36.491802 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 19:36:36.835003 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:36.834971 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnf7m/must-gather-d7hch" event={"ID":"93f54cc4-7c15-4623-8b33-1b96d26ee470","Type":"ContainerStarted","Data":"4def1462e4f30acc5eb324bd3ceca7376443bf2625b3bf4807d11f6dbf1f4c67"} Sep 16 19:36:43.872127 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:43.872085 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnf7m/must-gather-d7hch" event={"ID":"93f54cc4-7c15-4623-8b33-1b96d26ee470","Type":"ContainerStarted","Data":"a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b"} Sep 16 19:36:43.872570 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:43.872136 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnf7m/must-gather-d7hch" event={"ID":"93f54cc4-7c15-4623-8b33-1b96d26ee470","Type":"ContainerStarted","Data":"be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab"} Sep 16 19:36:43.889769 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:36:43.889715 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dnf7m/must-gather-d7hch" podStartSLOduration=2.635478501 podStartE2EDuration="8.889696449s" totalImagesPullingTime="6.254217948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:36:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:36:43.888213933 +0000 UTC m=+3064.211906782" watchObservedRunningTime="2026-09-16 19:36:43.889696449 +0000 UTC m=+3064.213389298" Sep 16 19:37:05.971286 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:05.971249 2563 generic.go:356] "Generic (PLEG): container finished" podID="93f54cc4-7c15-4623-8b33-1b96d26ee470" containerID="a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b" exitCode=0 Sep 16 19:37:05.971719 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:05.971324 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dnf7m/must-gather-d7hch" event={"ID":"93f54cc4-7c15-4623-8b33-1b96d26ee470","Type":"ContainerDied","Data":"a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b"} Sep 16 19:37:05.971719 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:05.971658 2563 scope.go:117] "RemoveContainer" containerID="a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b" Sep 16 19:37:06.816066 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:06.816035 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dnf7m_must-gather-d7hch_93f54cc4-7c15-4623-8b33-1b96d26ee470/gather/0.log" Sep 16 19:37:10.042592 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:10.042558 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-klhb5_0fed9f62-c1e7-419f-b5d2-85d3214e9d8e/global-pull-secret-syncer/0.log" Sep 16 19:37:10.181435 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:10.181381 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-m5czp_f741ffaa-797b-4f41-ac31-6da5fe63d9bd/konnectivity-agent/0.log" Sep 16 19:37:10.239939 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:10.239906 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-125.ec2.internal_153bf23bf7549998ce569e477f0251ee/haproxy/0.log" Sep 16 19:37:12.235738 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:12.235698 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-dnf7m/must-gather-d7hch"] Sep 16 19:37:12.236267 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:12.235903 2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-dnf7m/must-gather-d7hch" podUID="93f54cc4-7c15-4623-8b33-1b96d26ee470" containerName="copy" containerID="cri-o://be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab" gracePeriod=2 Sep 16 19:37:12.238002 ip-10-0-134-125 kubenswrapper[2563]: E0916 19:37:12.237968 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-d7hch\" is forbidden: User \"system:node:ip-10-0-134-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dnf7m\": no relationship found between node 'ip-10-0-134-125.ec2.internal' and this object" podUID="93f54cc4-7c15-4623-8b33-1b96d26ee470" pod="openshift-must-gather-dnf7m/must-gather-d7hch" Sep 16 19:37:12.239204 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:12.239179 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-dnf7m/must-gather-d7hch"] Sep 16 19:37:12.467720 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:12.467696 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dnf7m_must-gather-d7hch_93f54cc4-7c15-4623-8b33-1b96d26ee470/copy/0.log" Sep 16 19:37:12.469321 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:12.469303 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnf7m/must-gather-d7hch" Sep 16 19:37:12.605062 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:12.605018 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93f54cc4-7c15-4623-8b33-1b96d26ee470-must-gather-output\") pod \"93f54cc4-7c15-4623-8b33-1b96d26ee470\" (UID: \"93f54cc4-7c15-4623-8b33-1b96d26ee470\") " Sep 16 19:37:12.605255 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:12.605084 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbq5r\" (UniqueName: \"kubernetes.io/projected/93f54cc4-7c15-4623-8b33-1b96d26ee470-kube-api-access-pbq5r\") pod \"93f54cc4-7c15-4623-8b33-1b96d26ee470\" (UID: \"93f54cc4-7c15-4623-8b33-1b96d26ee470\") " Sep 16 19:37:12.606580 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:12.606552 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f54cc4-7c15-4623-8b33-1b96d26ee470-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "93f54cc4-7c15-4623-8b33-1b96d26ee470" (UID: "93f54cc4-7c15-4623-8b33-1b96d26ee470"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 19:37:12.607334 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:12.607311 2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f54cc4-7c15-4623-8b33-1b96d26ee470-kube-api-access-pbq5r" (OuterVolumeSpecName: "kube-api-access-pbq5r") pod "93f54cc4-7c15-4623-8b33-1b96d26ee470" (UID: "93f54cc4-7c15-4623-8b33-1b96d26ee470"). InnerVolumeSpecName "kube-api-access-pbq5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 19:37:12.706083 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:12.706047 2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/93f54cc4-7c15-4623-8b33-1b96d26ee470-must-gather-output\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 19:37:12.706083 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:12.706075 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbq5r\" (UniqueName: \"kubernetes.io/projected/93f54cc4-7c15-4623-8b33-1b96d26ee470-kube-api-access-pbq5r\") on node \"ip-10-0-134-125.ec2.internal\" DevicePath \"\"" Sep 16 19:37:13.001027 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:13.000954 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dnf7m_must-gather-d7hch_93f54cc4-7c15-4623-8b33-1b96d26ee470/copy/0.log" Sep 16 19:37:13.001309 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:13.001275 2563 generic.go:356] "Generic (PLEG): container finished" podID="93f54cc4-7c15-4623-8b33-1b96d26ee470" containerID="be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab" exitCode=143 Sep 16 19:37:13.001406 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:13.001328 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dnf7m/must-gather-d7hch" Sep 16 19:37:13.001406 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:13.001394 2563 scope.go:117] "RemoveContainer" containerID="be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab" Sep 16 19:37:13.010034 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:13.010015 2563 scope.go:117] "RemoveContainer" containerID="a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b" Sep 16 19:37:13.022341 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:13.022317 2563 scope.go:117] "RemoveContainer" containerID="be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab" Sep 16 19:37:13.022667 ip-10-0-134-125 kubenswrapper[2563]: E0916 19:37:13.022648 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab\": container with ID starting with be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab not found: ID does not exist" containerID="be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab" Sep 16 19:37:13.022724 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:13.022677 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab"} err="failed to get container status \"be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab\": rpc error: code = NotFound desc = could not find container \"be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab\": container with ID starting with be000fc0dc754871161aa47d171dc7190623c4abcb32cdcb985ffed4a98e54ab not found: ID does not exist" Sep 16 19:37:13.022724 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:13.022696 2563 scope.go:117] "RemoveContainer" containerID="a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b" Sep 16 19:37:13.022987 ip-10-0-134-125 kubenswrapper[2563]: E0916 19:37:13.022965 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b\": container with ID starting with a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b not found: ID does not exist" containerID="a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b" Sep 16 19:37:13.023071 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:13.022992 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b"} err="failed to get container status \"a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b\": rpc error: code = NotFound desc = could not find container \"a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b\": container with ID starting with a4c2c68dbafb04a282e62e0061d5f5d35a87eaa75567e1c40b2d9057114f647b not found: ID does not exist" Sep 16 19:37:14.210355 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.210326 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-hms5p_4f1e811d-0e94-42b6-888c-3e2f1952434c/cluster-monitoring-operator/0.log" Sep 16 19:37:14.242237 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.242220 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-k6mjb_40efaf55-090f-49b5-97fb-c3a3fef1e69f/kube-state-metrics/0.log" Sep 16 19:37:14.269891 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.269868 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-k6mjb_40efaf55-090f-49b5-97fb-c3a3fef1e69f/kube-rbac-proxy-main/0.log" Sep 16 19:37:14.296926 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.296902 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-k6mjb_40efaf55-090f-49b5-97fb-c3a3fef1e69f/kube-rbac-proxy-self/0.log" Sep 16 19:37:14.335613 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.335583 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93f54cc4-7c15-4623-8b33-1b96d26ee470" path="/var/lib/kubelet/pods/93f54cc4-7c15-4623-8b33-1b96d26ee470/volumes" Sep 16 19:37:14.527527 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.527501 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pdkl8_70580a6d-7e35-4eb0-85eb-65239da83d0b/node-exporter/0.log" Sep 16 19:37:14.554947 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.554919 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pdkl8_70580a6d-7e35-4eb0-85eb-65239da83d0b/kube-rbac-proxy/0.log" Sep 16 19:37:14.584013 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.583991 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pdkl8_70580a6d-7e35-4eb0-85eb-65239da83d0b/init-textfile/0.log" Sep 16 19:37:14.820317 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.820232 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92d7889a-6c53-4354-8c2e-76f4b64c20b3/prometheus/0.log" Sep 16 19:37:14.848535 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.848511 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92d7889a-6c53-4354-8c2e-76f4b64c20b3/config-reloader/0.log" Sep 16 19:37:14.878978 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.878954 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92d7889a-6c53-4354-8c2e-76f4b64c20b3/thanos-sidecar/0.log" Sep 16 19:37:14.906491 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.906462 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92d7889a-6c53-4354-8c2e-76f4b64c20b3/kube-rbac-proxy-web/0.log" Sep 16 19:37:14.928944 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.928924 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92d7889a-6c53-4354-8c2e-76f4b64c20b3/kube-rbac-proxy/0.log" Sep 16 19:37:14.952037 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.952012 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92d7889a-6c53-4354-8c2e-76f4b64c20b3/kube-rbac-proxy-thanos/0.log" Sep 16 19:37:14.975555 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:14.975533 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_92d7889a-6c53-4354-8c2e-76f4b64c20b3/init-config-reloader/0.log" Sep 16 19:37:15.009305 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:15.009273 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-xgwks_a47d1919-84c7-4649-af42-ed5e2c37c05a/prometheus-operator/0.log" Sep 16 19:37:15.041049 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:15.041028 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-xgwks_a47d1919-84c7-4649-af42-ed5e2c37c05a/kube-rbac-proxy/0.log" Sep 16 19:37:15.192051 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:15.192025 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/thanos-query/0.log" Sep 16 19:37:15.213848 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:15.213804 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/kube-rbac-proxy-web/0.log" Sep 16 19:37:15.246971 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:15.246941 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/kube-rbac-proxy/0.log" Sep 16 19:37:15.270341 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:15.270311 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/prom-label-proxy/0.log" Sep 16 19:37:15.292494 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:15.292448 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/kube-rbac-proxy-rules/0.log" Sep 16 19:37:15.314360 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:15.314335 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-766d64ccfc-wzr59_0c907ae1-316c-4570-b5de-181d7dd64b61/kube-rbac-proxy-metrics/0.log" Sep 16 19:37:16.648756 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:16.648711 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-nqzzv_7b5367e2-4fb0-43ec-8046-f1fdf490abeb/networking-console-plugin/0.log" Sep 16 19:37:17.178967 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:17.178935 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/2.log" Sep 16 19:37:17.183671 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:17.183647 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-f2hw9_322e4d63-e09c-4b27-8681-32b11e13c0cf/console-operator/3.log" Sep 16 19:37:18.308868 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.308820 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4"] Sep 16 19:37:18.309249 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.309139 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f54cc4-7c15-4623-8b33-1b96d26ee470" containerName="copy" Sep 16 19:37:18.309249 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.309150 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f54cc4-7c15-4623-8b33-1b96d26ee470" containerName="copy" Sep 16 19:37:18.309249 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.309182 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f54cc4-7c15-4623-8b33-1b96d26ee470" containerName="gather" Sep 16 19:37:18.309249 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.309188 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f54cc4-7c15-4623-8b33-1b96d26ee470" containerName="gather" Sep 16 19:37:18.309249 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.309241 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="93f54cc4-7c15-4623-8b33-1b96d26ee470" containerName="gather" Sep 16 19:37:18.309249 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.309249 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="93f54cc4-7c15-4623-8b33-1b96d26ee470" containerName="copy" Sep 16 19:37:18.312917 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.312895 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.314963 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.314940 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grqt8\"/\"openshift-service-ca.crt\"" Sep 16 19:37:18.315584 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.315567 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-grqt8\"/\"default-dockercfg-k5d84\"" Sep 16 19:37:18.315681 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.315581 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-grqt8\"/\"kube-root-ca.crt\"" Sep 16 19:37:18.326151 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.326130 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4"] Sep 16 19:37:18.347864 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.347824 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-lib-modules\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.347994 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.347880 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-sys\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.347994 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.347897 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-proc\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.347994 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.347927 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwsql\" (UniqueName: \"kubernetes.io/projected/57540a61-d379-4e88-9f49-9a7b11e56825-kube-api-access-wwsql\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.347994 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.347968 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-podres\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.449143 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.449099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-sys\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.449143 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.449139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-proc\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.449411 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.449157 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwsql\" (UniqueName: \"kubernetes.io/projected/57540a61-d379-4e88-9f49-9a7b11e56825-kube-api-access-wwsql\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.449411 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.449206 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-podres\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.449411 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.449240 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-sys\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.449411 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.449261 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-lib-modules\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.449411 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.449243 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-proc\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.449411 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.449407 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-podres\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.449659 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.449415 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57540a61-d379-4e88-9f49-9a7b11e56825-lib-modules\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.457202 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.457173 2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwsql\" (UniqueName: \"kubernetes.io/projected/57540a61-d379-4e88-9f49-9a7b11e56825-kube-api-access-wwsql\") pod \"perf-node-gather-daemonset-2t8z4\" (UID: \"57540a61-d379-4e88-9f49-9a7b11e56825\") " pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.622380 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.622289 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:18.896744 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:18.896714 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4"] Sep 16 19:37:18.899685 ip-10-0-134-125 kubenswrapper[2563]: W0916 19:37:18.899655 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod57540a61_d379_4e88_9f49_9a7b11e56825.slice/crio-5ce9a431bcad4e649bb8d302358f4b43165d59ec6b0d0e4e3f54b9762aa319c6 WatchSource:0}: Error finding container 5ce9a431bcad4e649bb8d302358f4b43165d59ec6b0d0e4e3f54b9762aa319c6: Status 404 returned error can't find the container with id 5ce9a431bcad4e649bb8d302358f4b43165d59ec6b0d0e4e3f54b9762aa319c6 Sep 16 19:37:19.026051 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:19.026012 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" event={"ID":"57540a61-d379-4e88-9f49-9a7b11e56825","Type":"ContainerStarted","Data":"2ad32be7f286161145dddbd9f648e4c125012bb59993a67c4078275fe06ea436"} Sep 16 19:37:19.026249 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:19.026072 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" event={"ID":"57540a61-d379-4e88-9f49-9a7b11e56825","Type":"ContainerStarted","Data":"5ce9a431bcad4e649bb8d302358f4b43165d59ec6b0d0e4e3f54b9762aa319c6"} Sep 16 19:37:19.026249 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:19.026107 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:19.047134 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:19.047074 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" podStartSLOduration=1.047054522 podStartE2EDuration="1.047054522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:37:19.045292641 +0000 UTC m=+3099.368985482" watchObservedRunningTime="2026-09-16 19:37:19.047054522 +0000 UTC m=+3099.370747370" Sep 16 19:37:19.251201 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:19.251132 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xrfb7_b1ceca87-379c-489d-b13f-6365d0a612b2/dns/0.log" Sep 16 19:37:19.273179 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:19.273150 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xrfb7_b1ceca87-379c-489d-b13f-6365d0a612b2/kube-rbac-proxy/0.log" Sep 16 19:37:19.321456 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:19.321424 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lh5wh_f2231582-bd0c-440b-9841-bd4740794c8c/dns-node-resolver/0.log" Sep 16 19:37:19.845330 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:19.845301 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h946g_01d77fab-c94e-46ea-b0e8-1b2b02f5d977/node-ca/0.log" Sep 16 19:37:20.651805 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:20.651765 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-95c96d4d4-gg962_2a1abe11-738f-42b1-be3c-fbd279e38659/router/0.log" Sep 16 19:37:20.998532 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:20.998443 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8vrw2_d1bc280d-ee46-48fe-a939-7b6d82bb1c92/serve-healthcheck-canary/0.log" Sep 16 19:37:21.567735 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:21.567691 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wdg6x_1c8b0805-a667-43e8-9a6e-2fef68669628/kube-rbac-proxy/0.log" Sep 16 19:37:21.588521 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:21.588490 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wdg6x_1c8b0805-a667-43e8-9a6e-2fef68669628/exporter/0.log" Sep 16 19:37:21.611934 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:21.611894 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wdg6x_1c8b0805-a667-43e8-9a6e-2fef68669628/extractor/0.log" Sep 16 19:37:24.098809 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:24.098778 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hmw4n_b01050a3-2637-4a00-988c-fd0248a72ec3/s3-init/0.log" Sep 16 19:37:24.117868 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:24.117820 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hmw4n_b01050a3-2637-4a00-988c-fd0248a72ec3/download-hf-model/0.log" Sep 16 19:37:25.039464 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:25.039429 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-grqt8/perf-node-gather-daemonset-2t8z4" Sep 16 19:37:28.462295 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:28.462211 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-z9cfw_8c28b886-8c19-4946-9dba-8930bf946190/migrator/0.log" Sep 16 19:37:28.484438 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:28.484399 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-z9cfw_8c28b886-8c19-4946-9dba-8930bf946190/graceful-termination/0.log" Sep 16 19:37:28.803515 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:28.803479 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-4rpwl_492c472f-5f87-465e-9534-058f606e5944/kube-storage-version-migrator-operator/1.log" Sep 16 19:37:28.804519 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:28.804499 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-4rpwl_492c472f-5f87-465e-9534-058f606e5944/kube-storage-version-migrator-operator/0.log" Sep 16 19:37:29.951050 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:29.951021 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjvnd_003ff6ca-4a01-4b09-98ea-17bf09a51ffe/kube-multus-additional-cni-plugins/0.log" Sep 16 19:37:30.004979 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:30.004950 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjvnd_003ff6ca-4a01-4b09-98ea-17bf09a51ffe/egress-router-binary-copy/0.log" Sep 16 19:37:30.027877 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:30.027844 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjvnd_003ff6ca-4a01-4b09-98ea-17bf09a51ffe/cni-plugins/0.log" Sep 16 19:37:30.053586 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:30.053555 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjvnd_003ff6ca-4a01-4b09-98ea-17bf09a51ffe/bond-cni-plugin/0.log" Sep 16 19:37:30.076028 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:30.075998 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjvnd_003ff6ca-4a01-4b09-98ea-17bf09a51ffe/routeoverride-cni/0.log" Sep 16 19:37:30.098101 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:30.098072 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjvnd_003ff6ca-4a01-4b09-98ea-17bf09a51ffe/whereabouts-cni-bincopy/0.log" Sep 16 19:37:30.119529 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:30.119488 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vjvnd_003ff6ca-4a01-4b09-98ea-17bf09a51ffe/whereabouts-cni/0.log" Sep 16 19:37:30.323753 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:30.323715 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jprzs_97e78433-797a-47b6-9b23-798ddb52e025/kube-multus/0.log" Sep 16 19:37:30.379076 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:30.379046 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lq68v_650deff9-b845-4ef3-acad-6bdee9667025/network-metrics-daemon/0.log" Sep 16 19:37:30.400891 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:30.400851 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-lq68v_650deff9-b845-4ef3-acad-6bdee9667025/kube-rbac-proxy/0.log" Sep 16 19:37:31.228076 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:31.228047 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx8nl_4ae005ac-ab99-433a-bd5c-7b8193356ffb/ovn-controller/0.log" Sep 16 19:37:31.249742 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:31.249710 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx8nl_4ae005ac-ab99-433a-bd5c-7b8193356ffb/ovn-acl-logging/0.log" Sep 16 19:37:31.273127 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:31.273100 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx8nl_4ae005ac-ab99-433a-bd5c-7b8193356ffb/kube-rbac-proxy-node/0.log" Sep 16 19:37:31.294243 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:31.294212 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx8nl_4ae005ac-ab99-433a-bd5c-7b8193356ffb/kube-rbac-proxy-ovn-metrics/0.log" Sep 16 19:37:31.312729 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:31.312695 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx8nl_4ae005ac-ab99-433a-bd5c-7b8193356ffb/northd/0.log" Sep 16 19:37:31.333565 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:31.333534 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx8nl_4ae005ac-ab99-433a-bd5c-7b8193356ffb/nbdb/0.log" Sep 16 19:37:31.357242 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:31.357213 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx8nl_4ae005ac-ab99-433a-bd5c-7b8193356ffb/sbdb/0.log" Sep 16 19:37:31.394453 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:31.394417 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bx8nl_4ae005ac-ab99-433a-bd5c-7b8193356ffb/ovnkube-controller/0.log" Sep 16 19:37:32.792848 ip-10-0-134-125 kubenswrapper[2563]: I0916 19:37:32.792805 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9k5bx_94040f40-6ebc-4c73-9280-83b5e973a840/network-check-target-container/0.log"