Apr 16 16:00:19.740104 ip-10-0-143-177 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Apr 16 16:00:19.740116 ip-10-0-143-177 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Apr 16 16:00:19.740125 ip-10-0-143-177 systemd[1]: kubelet.service: Failed with result 'resources'. Apr 16 16:00:19.740424 ip-10-0-143-177 systemd[1]: Failed to start Kubernetes Kubelet. Apr 16 16:00:29.985291 ip-10-0-143-177 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Apr 16 16:00:29.985308 ip-10-0-143-177 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6c38b21b40444301bfcb5a34198dea78 -- Apr 16 16:02:41.596326 ip-10-0-143-177 systemd[1]: Starting Kubernetes Kubelet... Apr 16 16:02:42.149900 ip-10-0-143-177 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 16:02:42.149900 ip-10-0-143-177 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Apr 16 16:02:42.149900 ip-10-0-143-177 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 16:02:42.149900 ip-10-0-143-177 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Apr 16 16:02:42.149900 ip-10-0-143-177 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Apr 16 16:02:42.151081 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.150884 2576 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Apr 16 16:02:42.154842 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154819 2576 feature_gate.go:328] unrecognized feature gate: Example Apr 16 16:02:42.154842 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154839 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 16:02:42.154842 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154843 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 16:02:42.154842 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154848 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 16:02:42.154842 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154851 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154854 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154857 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154860 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154863 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154866 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154869 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154872 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154874 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154877 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154880 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154883 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154885 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154888 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154891 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154894 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154896 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154899 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154909 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154912 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 16:02:42.155057 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154915 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154917 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154920 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154922 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154925 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154927 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154931 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154935 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154939 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154944 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154947 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154950 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154953 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154956 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154960 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154964 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154966 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154969 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 16:02:42.155537 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154972 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154974 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154977 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154979 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154982 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154984 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154987 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154990 2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154992 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154995 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.154997 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155000 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155003 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155006 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155009 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155029 2576 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155034 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155038 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155041 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155044 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 16:02:42.156008 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155046 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155050 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155053 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155056 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155059 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155062 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155065 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155068 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155071 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155074 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155077 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155080 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155082 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155085 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155088 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155090 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155095 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155098 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155101 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155103 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 16:02:42.156517 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155106 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155108 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155111 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155114 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155561 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155567 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155570 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155573 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155577 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155581 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155583 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155586 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155588 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155591 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155594 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155596 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155599 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155601 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155604 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 16:02:42.156994 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155606 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155610 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155613 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155615 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155618 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155621 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155624 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155627 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155630 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155633 2576 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155636 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155638 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155641 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155643 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155647 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155651 2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155656 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155661 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155664 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 16:02:42.157476 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155667 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155670 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155673 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155676 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155679 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155682 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155684 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155687 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155690 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155693 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155697 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155700 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155704 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155706 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155710 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155713 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155716 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155718 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155721 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155724 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 16:02:42.157947 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155726 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155729 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155731 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155735 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155737 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155740 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155743 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155745 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155748 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155750 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155753 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155755 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155757 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155760 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155762 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155765 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155768 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155771 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155773 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155776 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 16:02:42.158454 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155779 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155781 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155783 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155786 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155788 2576 feature_gate.go:328] unrecognized feature gate: Example Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155793 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155796 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155799 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155801 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155804 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155806 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.155809 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157179 2576 flags.go:64] FLAG: --address="0.0.0.0" Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157189 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157197 2576 flags.go:64] FLAG: --anonymous-auth="true" Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157202 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157213 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157216 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157220 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157225 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157229 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Apr 16 16:02:42.158944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157232 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157235 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157238 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157241 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157244 2576 flags.go:64] FLAG: --cgroup-root="" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157248 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157251 2576 flags.go:64] FLAG: --client-ca-file="" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157254 2576 flags.go:64] FLAG: --cloud-config="" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157256 2576 flags.go:64] FLAG: --cloud-provider="external" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157259 2576 flags.go:64] FLAG: --cluster-dns="[]" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157266 2576 flags.go:64] FLAG: --cluster-domain="" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157268 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157272 2576 flags.go:64] FLAG: --config-dir="" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157274 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157278 2576 flags.go:64] FLAG: --container-log-max-files="5" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157282 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157286 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157289 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157293 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157295 2576 flags.go:64] FLAG: --contention-profiling="false" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157298 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157301 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157304 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157307 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157311 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Apr 16 16:02:42.159486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157314 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157317 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157320 2576 flags.go:64] FLAG: --enable-load-reader="false" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157334 2576 flags.go:64] FLAG: --enable-server="true" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157338 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157344 2576 flags.go:64] FLAG: --event-burst="100" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157347 2576 flags.go:64] FLAG: --event-qps="50" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157349 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157352 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157355 2576 flags.go:64] FLAG: --eviction-hard="" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157359 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157362 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157365 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157368 2576 flags.go:64] FLAG: --eviction-soft="" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157371 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157374 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157377 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157379 2576 flags.go:64] FLAG: --experimental-mounter-path="" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157382 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157385 2576 flags.go:64] FLAG: --fail-swap-on="true" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157388 2576 flags.go:64] FLAG: --feature-gates="" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157392 2576 flags.go:64] FLAG: --file-check-frequency="20s" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157395 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157398 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157401 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157405 2576 flags.go:64] FLAG: --healthz-port="10248" Apr 16 16:02:42.160096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157408 2576 flags.go:64] FLAG: --help="false" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157411 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157414 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157417 2576 flags.go:64] FLAG: --http-check-frequency="20s" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157420 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157423 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157427 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157429 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157432 2576 flags.go:64] FLAG: --image-service-endpoint="" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157435 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157438 2576 flags.go:64] FLAG: --kube-api-burst="100" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157441 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157445 2576 flags.go:64] FLAG: --kube-api-qps="50" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157447 2576 flags.go:64] FLAG: --kube-reserved="" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157450 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157453 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157456 2576 flags.go:64] FLAG: --kubelet-cgroups="" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157459 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157462 2576 flags.go:64] FLAG: --lock-file="" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157465 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157468 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157471 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157477 2576 flags.go:64] FLAG: --log-json-split-stream="false" Apr 16 16:02:42.160729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157480 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157483 2576 flags.go:64] FLAG: --log-text-split-stream="false" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157486 2576 flags.go:64] FLAG: --logging-format="text" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157489 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157492 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157495 2576 flags.go:64] FLAG: --manifest-url="" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157498 2576 flags.go:64] FLAG: --manifest-url-header="" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157503 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157506 2576 flags.go:64] FLAG: --max-open-files="1000000" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157510 2576 flags.go:64] FLAG: --max-pods="110" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157514 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157517 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157520 2576 flags.go:64] FLAG: --memory-manager-policy="None" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157523 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157526 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157528 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157532 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157539 2576 flags.go:64] FLAG: --node-status-max-images="50" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157556 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157560 2576 flags.go:64] FLAG: --oom-score-adj="-999" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157564 2576 flags.go:64] FLAG: --pod-cidr="" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157567 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:dc76bab72f320de3d4105c90d73c4fb139c09e20ce0fa8dcbc0cb59920d27dec" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157572 2576 flags.go:64] FLAG: --pod-manifest-path="" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157576 2576 flags.go:64] FLAG: --pod-max-pids="-1" Apr 16 16:02:42.161325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157579 2576 flags.go:64] FLAG: --pods-per-core="0" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157582 2576 flags.go:64] FLAG: --port="10250" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157585 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157588 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04f889ca10fd6b0b3" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157591 2576 flags.go:64] FLAG: --qos-reserved="" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157594 2576 flags.go:64] FLAG: --read-only-port="10255" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157597 2576 flags.go:64] FLAG: --register-node="true" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157600 2576 flags.go:64] FLAG: --register-schedulable="true" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157603 2576 flags.go:64] FLAG: --register-with-taints="" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157607 2576 flags.go:64] FLAG: --registry-burst="10" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157610 2576 flags.go:64] FLAG: --registry-qps="5" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157613 2576 flags.go:64] FLAG: --reserved-cpus="" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157616 2576 flags.go:64] FLAG: --reserved-memory="" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157619 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157623 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157626 2576 flags.go:64] FLAG: --rotate-certificates="false" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157629 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157631 2576 flags.go:64] FLAG: --runonce="false" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157634 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157638 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157640 2576 flags.go:64] FLAG: --seccomp-default="false" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157643 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157646 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157649 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157652 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157656 2576 flags.go:64] FLAG: --storage-driver-password="root" Apr 16 16:02:42.161990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157659 2576 flags.go:64] FLAG: --storage-driver-secure="false" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157661 2576 flags.go:64] FLAG: --storage-driver-table="stats" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157664 2576 flags.go:64] FLAG: --storage-driver-user="root" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157667 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157671 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157673 2576 flags.go:64] FLAG: --system-cgroups="" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157677 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157682 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157685 2576 flags.go:64] FLAG: --tls-cert-file="" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157688 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157692 2576 flags.go:64] FLAG: --tls-min-version="" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157694 2576 flags.go:64] FLAG: --tls-private-key-file="" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157697 2576 flags.go:64] FLAG: --topology-manager-policy="none" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157700 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157707 2576 flags.go:64] FLAG: --topology-manager-scope="container" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157710 2576 flags.go:64] FLAG: --v="2" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157714 2576 flags.go:64] FLAG: --version="false" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157718 2576 flags.go:64] FLAG: --vmodule="" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157723 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.157726 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157830 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157835 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157839 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 16:02:42.162690 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157842 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157845 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157848 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157851 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157853 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157856 2576 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157859 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157862 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157865 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157868 2576 feature_gate.go:328] unrecognized feature gate: Example Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157870 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157873 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157875 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157878 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157881 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157883 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157886 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157888 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157891 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157893 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 16:02:42.163268 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157896 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157898 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157901 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157905 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157908 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157911 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157913 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157916 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157918 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157922 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157925 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157927 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157930 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157932 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157935 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157937 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157940 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157943 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157945 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157948 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 16:02:42.163778 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157950 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157953 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157956 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157960 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157963 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157966 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157971 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157974 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157977 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157980 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157982 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157985 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157988 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157990 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157993 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157997 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.157999 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158002 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158005 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 16:02:42.164290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158007 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158010 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158030 2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158033 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158035 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158038 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158041 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158043 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158046 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158048 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158051 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158053 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158056 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158058 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158061 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158064 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158067 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158069 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158071 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158074 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 16:02:42.164775 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158078 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 16:02:42.165290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158081 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 16:02:42.165290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158083 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 16:02:42.165290 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.158085 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 16:02:42.165290 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.158991 2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 16:02:42.166776 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.166756 2576 server.go:530] "Kubelet version" kubeletVersion="v1.33.9" Apr 16 16:02:42.166817 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.166777 2576 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Apr 16 16:02:42.166847 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166837 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 16:02:42.166847 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166844 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 16:02:42.166847 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166847 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166852 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166856 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166860 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166863 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166866 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166869 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166871 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166874 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166877 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166879 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166882 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166885 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166888 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166890 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166893 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166895 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166899 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166901 2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166904 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 16:02:42.166927 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166907 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166909 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166912 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166915 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166918 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166920 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166925 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166929 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166932 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166937 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166940 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166943 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166946 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166949 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166952 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166955 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166957 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166960 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166962 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 16:02:42.167509 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166965 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166968 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166970 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166973 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166976 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166978 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166981 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166983 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166986 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166989 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166991 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166994 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166996 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.166999 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167001 2576 feature_gate.go:328] unrecognized feature gate: Example Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167004 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167007 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167027 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167030 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167033 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 16:02:42.167968 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167036 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167038 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167041 2576 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167044 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167047 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167050 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167053 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167055 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167058 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167061 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167064 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167066 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167069 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167072 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167074 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167077 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167079 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167082 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167085 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 16:02:42.168484 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167088 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167090 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167093 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167095 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167098 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167101 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.167106 2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167244 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167250 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167254 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167256 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167260 2576 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167263 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167265 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167268 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Apr 16 16:02:42.168941 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167271 2576 feature_gate.go:328] unrecognized feature gate: Example2 Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167274 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167277 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167280 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167282 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167285 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167288 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167290 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167293 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167296 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167298 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167301 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167303 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167306 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167308 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167311 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167314 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167317 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167319 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167322 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Apr 16 16:02:42.169346 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167325 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167327 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167330 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167332 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167335 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167337 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167341 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167343 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167354 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167358 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167363 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167366 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167370 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167372 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167375 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167379 2576 feature_gate.go:328] unrecognized feature gate: Example Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167382 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167384 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167387 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167389 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Apr 16 16:02:42.169826 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167392 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167395 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167397 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167400 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167402 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167405 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167407 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167410 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167412 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167415 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167417 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167420 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167422 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167425 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167427 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167430 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167432 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167435 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167438 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Apr 16 16:02:42.170322 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167442 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167445 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167449 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167452 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167454 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167457 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167460 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167463 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167465 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167469 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167471 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167474 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167476 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167479 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167482 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167484 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167487 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167489 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Apr 16 16:02:42.170791 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:42.167492 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Apr 16 16:02:42.171247 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.167496 2576 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} Apr 16 16:02:42.171247 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.168271 2576 server.go:962] "Client rotation is on, will bootstrap in background" Apr 16 16:02:42.171247 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.170341 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Apr 16 16:02:42.171333 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.171321 2576 server.go:1019] "Starting client certificate rotation" Apr 16 16:02:42.171433 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.171414 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 16 16:02:42.172141 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.172116 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Apr 16 16:02:42.203490 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.203467 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 16 16:02:42.206234 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.206216 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Apr 16 16:02:42.225064 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.225039 2576 log.go:25] "Validated CRI v1 runtime API" Apr 16 16:02:42.231231 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.231216 2576 log.go:25] "Validated CRI v1 image API" Apr 16 16:02:42.233126 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.233106 2576 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Apr 16 16:02:42.236002 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.235982 2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 16 16:02:42.238198 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.238179 2576 fs.go:135] Filesystem UUIDs: map[3144d156-f43c-40ea-8dc7-2f8f102ec117:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f7bc31a2-3c6c-4767-a307-0d67f1456cda:/dev/nvme0n1p4] Apr 16 16:02:42.238253 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.238199 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Apr 16 16:02:42.244303 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.244171 2576 manager.go:217] Machine: {Timestamp:2026-04-16 16:02:42.241928389 +0000 UTC m=+0.500675210 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099794 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec248a483a4520eb113e98300cc2d8bc SystemUUID:ec248a48-3a45-20eb-113e-98300cc2d8bc BootID:6c38b21b-4044-4301-bfcb-5a34198dea78 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6098944 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4d:af:4e:51:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4d:af:4e:51:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:95:b1:5c:65:22 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Apr 16 16:02:42.244303 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.244298 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Apr 16 16:02:42.244425 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.244413 2576 manager.go:233] Version: {KernelVersion:5.14.0-570.104.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260401-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Apr 16 16:02:42.245797 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.245764 2576 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Apr 16 16:02:42.245977 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.245800 2576 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-177.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Apr 16 16:02:42.246083 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.245990 2576 topology_manager.go:138] "Creating topology manager with none policy" Apr 16 16:02:42.246083 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.246004 2576 container_manager_linux.go:306] "Creating device plugin manager" Apr 16 16:02:42.246083 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.246041 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 16 16:02:42.247028 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.247000 2576 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Apr 16 16:02:42.248795 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.248782 2576 state_mem.go:36] "Initialized new in-memory state store" Apr 16 16:02:42.248940 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.248928 2576 server.go:1267] "Using root directory" path="/var/lib/kubelet" Apr 16 16:02:42.251438 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.251425 2576 kubelet.go:491] "Attempting to sync node with API server" Apr 16 16:02:42.251495 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.251445 2576 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" Apr 16 16:02:42.251495 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.251462 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Apr 16 16:02:42.251495 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.251477 2576 kubelet.go:397] "Adding apiserver pod source" Apr 16 16:02:42.251495 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.251492 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Apr 16 16:02:42.252901 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.252885 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 16 16:02:42.252967 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.252910 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Apr 16 16:02:42.256747 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.256727 2576 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.10-2.rhaos4.20.gita4d0894.el9" apiVersion="v1" Apr 16 16:02:42.258794 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.258779 2576 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Apr 16 16:02:42.260576 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260557 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Apr 16 16:02:42.260576 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260574 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Apr 16 16:02:42.260667 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260580 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Apr 16 16:02:42.260667 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260586 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Apr 16 16:02:42.260667 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260594 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Apr 16 16:02:42.260667 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260602 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" Apr 16 16:02:42.260667 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260608 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Apr 16 16:02:42.260667 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260614 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Apr 16 16:02:42.260667 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260622 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" Apr 16 16:02:42.260667 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260628 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Apr 16 16:02:42.260667 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260642 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" Apr 16 16:02:42.260667 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.260651 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Apr 16 16:02:42.261911 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.261901 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" Apr 16 16:02:42.261911 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.261912 2576 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" Apr 16 16:02:42.263862 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.263837 2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Apr 16 16:02:42.263901 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.263837 2576 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Apr 16 16:02:42.265063 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.265049 2576 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-177.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Apr 16 16:02:42.265834 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.265822 2576 watchdog_linux.go:99] "Systemd watchdog is not enabled" Apr 16 16:02:42.265872 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.265858 2576 server.go:1295] "Started kubelet" Apr 16 16:02:42.265979 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.265941 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Apr 16 16:02:42.266042 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.265982 2576 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Apr 16 16:02:42.266080 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.266068 2576 server_v1.go:47] "podresources" method="list" useActivePods=true Apr 16 16:02:42.266681 ip-10-0-143-177 systemd[1]: Started Kubernetes Kubelet. Apr 16 16:02:42.267781 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.267751 2576 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Apr 16 16:02:42.269597 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.269579 2576 server.go:317] "Adding debug handlers to kubelet server" Apr 16 16:02:42.272745 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.272724 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Apr 16 16:02:42.273426 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.273411 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Apr 16 16:02:42.274310 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.274090 2576 desired_state_of_world_populator.go:150] "Desired state populator starts to run" Apr 16 16:02:42.274405 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.274186 2576 volume_manager.go:295] "The desired_state_of_world populator starts" Apr 16 16:02:42.274405 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.274380 2576 volume_manager.go:297] "Starting Kubelet Volume Manager" Apr 16 16:02:42.274647 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.274629 2576 reconstruct.go:97] "Volume reconstruction finished" Apr 16 16:02:42.274647 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.274646 2576 reconciler.go:26] "Reconciler: start to sync state" Apr 16 16:02:42.275031 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.275004 2576 factory.go:55] Registering systemd factory Apr 16 16:02:42.275031 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.275034 2576 factory.go:223] Registration of the systemd container factory successfully Apr 16 16:02:42.275351 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.275331 2576 factory.go:153] Registering CRI-O factory Apr 16 16:02:42.275510 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.275352 2576 factory.go:223] Registration of the crio container factory successfully Apr 16 16:02:42.275639 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.275625 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Apr 16 16:02:42.275699 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.275666 2576 factory.go:103] Registering Raw factory Apr 16 16:02:42.275699 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.275685 2576 manager.go:1196] Started watching for new ooms in manager Apr 16 16:02:42.276129 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.276090 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:42.277465 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.277229 2576 manager.go:319] Starting recovery of all containers Apr 16 16:02:42.279453 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.279432 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dlngn" Apr 16 16:02:42.285480 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.285444 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-177.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Apr 16 16:02:42.285883 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.285847 2576 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Apr 16 16:02:42.287415 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.285571 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-177.ec2.internal.18a6e1ca5ab724f7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-177.ec2.internal,UID:ip-10-0-143-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-177.ec2.internal,},FirstTimestamp:2026-04-16 16:02:42.265834743 +0000 UTC m=+0.524581544,LastTimestamp:2026-04-16 16:02:42.265834743 +0000 UTC m=+0.524581544,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-177.ec2.internal,}" Apr 16 16:02:42.288671 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.288657 2576 manager.go:324] Recovery completed Apr 16 16:02:42.290817 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.290795 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dlngn" Apr 16 16:02:42.292704 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.292690 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:02:42.295232 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.295213 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:02:42.295307 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.295245 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:02:42.295307 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.295257 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:02:42.295694 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.295682 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Apr 16 16:02:42.295694 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.295693 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Apr 16 16:02:42.295765 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.295708 2576 state_mem.go:36] "Initialized new in-memory state store" Apr 16 16:02:42.297053 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.296977 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-177.ec2.internal.18a6e1ca5c77b372 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-177.ec2.internal,UID:ip-10-0-143-177.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-177.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-177.ec2.internal,},FirstTimestamp:2026-04-16 16:02:42.295231346 +0000 UTC m=+0.553978147,LastTimestamp:2026-04-16 16:02:42.295231346 +0000 UTC m=+0.553978147,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-177.ec2.internal,}" Apr 16 16:02:42.297745 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.297733 2576 policy_none.go:49] "None policy: Start" Apr 16 16:02:42.297781 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.297750 2576 memory_manager.go:186] "Starting memorymanager" policy="None" Apr 16 16:02:42.297781 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.297760 2576 state_mem.go:35] "Initializing new in-memory state store" Apr 16 16:02:42.330732 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.330710 2576 manager.go:341] "Starting Device Plugin manager" Apr 16 16:02:42.343885 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.330745 2576 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Apr 16 16:02:42.343885 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.330756 2576 server.go:85] "Starting device plugin registration server" Apr 16 16:02:42.343885 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.331036 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Apr 16 16:02:42.343885 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.331048 2576 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Apr 16 16:02:42.343885 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.331148 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Apr 16 16:02:42.343885 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.331214 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Apr 16 16:02:42.343885 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.331222 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Apr 16 16:02:42.343885 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.332145 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Apr 16 16:02:42.343885 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.332255 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:42.412488 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.412403 2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" Apr 16 16:02:42.413673 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.413657 2576 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" Apr 16 16:02:42.413766 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.413686 2576 status_manager.go:230] "Starting to sync pod status with apiserver" Apr 16 16:02:42.413766 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.413710 2576 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Apr 16 16:02:42.413766 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.413720 2576 kubelet.go:2451] "Starting kubelet main sync loop" Apr 16 16:02:42.413766 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.413761 2576 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Apr 16 16:02:42.419358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.419333 2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 16:02:42.431285 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.431261 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:02:42.432261 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.432244 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:02:42.432363 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.432277 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:02:42.432363 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.432287 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:02:42.432363 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.432310 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.441193 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.441174 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.441294 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.441200 2576 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-177.ec2.internal\": node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:42.458618 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.458598 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:42.514432 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.514401 2576 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal"] Apr 16 16:02:42.514532 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.514483 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:02:42.515453 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.515437 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:02:42.515517 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.515472 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:02:42.515517 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.515487 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:02:42.516760 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.516748 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:02:42.516900 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.516886 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.516953 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.516916 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:02:42.517548 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.517534 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:02:42.517625 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.517560 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:02:42.517625 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.517570 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:02:42.517625 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.517538 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:02:42.517727 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.517630 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:02:42.517727 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.517641 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:02:42.519072 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.519054 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.519120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.519083 2576 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" Apr 16 16:02:42.519866 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.519841 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientMemory" Apr 16 16:02:42.519944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.519881 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasNoDiskPressure" Apr 16 16:02:42.519944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.519923 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeHasSufficientPID" Apr 16 16:02:42.549732 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.549706 2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-177.ec2.internal\" not found" node="ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.554060 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.554042 2576 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-177.ec2.internal\" not found" node="ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.559326 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.559309 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:42.575674 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.575649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c91eb3ac6a0e55d0bbe3b604fc28906-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"2c91eb3ac6a0e55d0bbe3b604fc28906\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.575767 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.575681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c91eb3ac6a0e55d0bbe3b604fc28906-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"2c91eb3ac6a0e55d0bbe3b604fc28906\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.575767 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.575708 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6c9c3cf395cb0ad946e6c0c60278a83-config\") pod \"kube-apiserver-proxy-ip-10-0-143-177.ec2.internal\" (UID: \"a6c9c3cf395cb0ad946e6c0c60278a83\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.660220 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.660172 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:42.676570 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.676516 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c91eb3ac6a0e55d0bbe3b604fc28906-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"2c91eb3ac6a0e55d0bbe3b604fc28906\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.676570 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.676549 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c91eb3ac6a0e55d0bbe3b604fc28906-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"2c91eb3ac6a0e55d0bbe3b604fc28906\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.676570 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.676569 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6c9c3cf395cb0ad946e6c0c60278a83-config\") pod \"kube-apiserver-proxy-ip-10-0-143-177.ec2.internal\" (UID: \"a6c9c3cf395cb0ad946e6c0c60278a83\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.676698 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.676599 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a6c9c3cf395cb0ad946e6c0c60278a83-config\") pod \"kube-apiserver-proxy-ip-10-0-143-177.ec2.internal\" (UID: \"a6c9c3cf395cb0ad946e6c0c60278a83\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.676698 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.676616 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2c91eb3ac6a0e55d0bbe3b604fc28906-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"2c91eb3ac6a0e55d0bbe3b604fc28906\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.676698 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.676624 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c91eb3ac6a0e55d0bbe3b604fc28906-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal\" (UID: \"2c91eb3ac6a0e55d0bbe3b604fc28906\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.760957 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.760916 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:42.853438 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.853402 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.856943 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:42.856928 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Apr 16 16:02:42.861431 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.861412 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:42.961922 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:42.961888 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:43.062493 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:43.062452 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:43.163001 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:43.162967 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:43.171145 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.171125 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Apr 16 16:02:43.171269 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.171254 2576 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Apr 16 16:02:43.264131 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:43.264053 2576 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-143-177.ec2.internal\" not found" Apr 16 16:02:43.273327 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.273309 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Apr 16 16:02:43.283191 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.283169 2576 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Apr 16 16:02:43.293048 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.293008 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-04-15 15:57:42 +0000 UTC" deadline="2028-01-14 08:46:07.166622998 +0000 UTC" Apr 16 16:02:43.293165 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.293049 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15304h43m23.873577868s" Apr 16 16:02:43.305448 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.305418 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-k7xwd" Apr 16 16:02:43.312032 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.311991 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-k7xwd" Apr 16 16:02:43.332763 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.332740 2576 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 16:02:43.373935 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.373907 2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" Apr 16 16:02:43.389241 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.389202 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 16 16:02:43.391394 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.391375 2576 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" Apr 16 16:02:43.398169 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.398143 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Apr 16 16:02:43.414183 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:43.414155 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6c9c3cf395cb0ad946e6c0c60278a83.slice/crio-e81818b7d015c25dca06b802525ac118be9aa114bef8ad5adc11ef395ad7a322 WatchSource:0}: Error finding container e81818b7d015c25dca06b802525ac118be9aa114bef8ad5adc11ef395ad7a322: Status 404 returned error can't find the container with id e81818b7d015c25dca06b802525ac118be9aa114bef8ad5adc11ef395ad7a322 Apr 16 16:02:43.415039 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:43.415001 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c91eb3ac6a0e55d0bbe3b604fc28906.slice/crio-a2b4f559cf45e920f7e224bed1482c64bb5092b40eb7e389581e07f783dd1f8a WatchSource:0}: Error finding container a2b4f559cf45e920f7e224bed1482c64bb5092b40eb7e389581e07f783dd1f8a: Status 404 returned error can't find the container with id a2b4f559cf45e920f7e224bed1482c64bb5092b40eb7e389581e07f783dd1f8a Apr 16 16:02:43.420617 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.420598 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 16:02:43.602388 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.602354 2576 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 16:02:43.816008 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:43.815936 2576 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 16:02:44.242178 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.242149 2576 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Apr 16 16:02:44.252335 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.252303 2576 apiserver.go:52] "Watching apiserver" Apr 16 16:02:44.261623 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.261495 2576 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Apr 16 16:02:44.263331 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.263301 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal","openshift-cluster-node-tuning-operator/tuned-q5jgq","openshift-network-diagnostics/network-check-target-2lh6g","openshift-network-operator/iptables-alerter-m98lv","openshift-ovn-kubernetes/ovnkube-node-gtjzw","kube-system/konnectivity-agent-lg294","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq","openshift-image-registry/node-ca-bgdhp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal","openshift-multus/multus-68jlg","openshift-multus/multus-additional-cni-plugins-dfnlw","openshift-multus/network-metrics-daemon-fpztb"] Apr 16 16:02:44.268032 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.267998 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.269983 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.269955 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:44.270180 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.270053 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:02:44.270180 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.270068 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m98lv" Apr 16 16:02:44.270502 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.270403 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Apr 16 16:02:44.270502 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.270461 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kd9ss\"" Apr 16 16:02:44.270712 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.270696 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Apr 16 16:02:44.272121 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.272102 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Apr 16 16:02:44.272251 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.272182 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Apr 16 16:02:44.272606 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.272570 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p2x9m\"" Apr 16 16:02:44.272698 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.272625 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Apr 16 16:02:44.274553 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.274500 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.274828 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.274753 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lg294" Apr 16 16:02:44.276848 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.276824 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.277369 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.277329 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Apr 16 16:02:44.277369 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.277370 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Apr 16 16:02:44.277557 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.277437 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Apr 16 16:02:44.277557 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.277490 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cm6fv\"" Apr 16 16:02:44.277557 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.277515 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Apr 16 16:02:44.277557 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.277522 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Apr 16 16:02:44.277730 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.277706 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-94srw\"" Apr 16 16:02:44.277779 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.277744 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Apr 16 16:02:44.277909 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.277864 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Apr 16 16:02:44.277909 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.277876 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Apr 16 16:02:44.279131 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.278985 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Apr 16 16:02:44.279131 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.279065 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Apr 16 16:02:44.279131 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.279118 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bgdhp" Apr 16 16:02:44.279561 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.279543 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Apr 16 16:02:44.279927 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.279910 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lpsvq\"" Apr 16 16:02:44.281257 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.281238 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Apr 16 16:02:44.281391 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.281370 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Apr 16 16:02:44.281485 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.281457 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fkwk5\"" Apr 16 16:02:44.281705 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.281687 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.281927 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.281909 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Apr 16 16:02:44.283916 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.283856 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hfdxs\"" Apr 16 16:02:44.284165 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284149 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.284517 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-run-systemd\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.284618 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284534 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-run-ovn\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.284618 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284565 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5460383-84d2-4bd9-aa8b-82bbace94cb8-konnectivity-ca\") pod \"konnectivity-agent-lg294\" (UID: \"b5460383-84d2-4bd9-aa8b-82bbace94cb8\") " pod="kube-system/konnectivity-agent-lg294" Apr 16 16:02:44.284618 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284592 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-device-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.284775 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-modprobe-d\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.284775 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284656 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Apr 16 16:02:44.284775 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284663 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-kubelet\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.284775 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284708 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Apr 16 16:02:44.284775 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284707 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-etc-openvswitch\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285026 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285026 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284801 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-ovnkube-script-lib\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285026 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.285026 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-registration-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.285026 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-sysctl-d\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.285026 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-sysctl-conf\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.285389 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284936 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Apr 16 16:02:44.285389 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-systemd-units\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285389 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.284892 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Apr 16 16:02:44.285389 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285186 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-var-lib-openvswitch\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285389 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285251 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-node-log\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285389 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-ovnkube-config\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285389 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csvfm\" (UniqueName: \"kubernetes.io/projected/0369ef86-452e-4148-ab71-6ccca93defc5-kube-api-access-csvfm\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.285389 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-systemd\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.285729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285396 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c0f9a7c-7995-43e1-9141-56467d871fce-tmp\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.285729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8bfj\" (UniqueName: \"kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj\") pod \"network-check-target-2lh6g\" (UID: \"23dd67cf-2731-4685-80f7-f4a1e75b6476\") " pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:44.285729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-slash\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285491 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-cni-netd\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285515 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-env-overrides\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-ovn-node-metrics-cert\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkk2q\" (UniqueName: \"kubernetes.io/projected/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-kube-api-access-nkk2q\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-kubernetes\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.285729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285684 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-log-socket\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.285729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-cni-bin\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-host\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285787 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5460383-84d2-4bd9-aa8b-82bbace94cb8-agent-certs\") pod \"konnectivity-agent-lg294\" (UID: \"b5460383-84d2-4bd9-aa8b-82bbace94cb8\") " pod="kube-system/konnectivity-agent-lg294" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285838 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-sys\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-lib-modules\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285930 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-tuned\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d44ec06b-81b1-4bac-9abd-8efadd98b94a-iptables-alerter-script\") pod \"iptables-alerter-m98lv\" (UID: \"d44ec06b-81b1-4bac-9abd-8efadd98b94a\") " pod="openshift-network-operator/iptables-alerter-m98lv" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.285974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d44ec06b-81b1-4bac-9abd-8efadd98b94a-host-slash\") pod \"iptables-alerter-m98lv\" (UID: \"d44ec06b-81b1-4bac-9abd-8efadd98b94a\") " pod="openshift-network-operator/iptables-alerter-m98lv" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.286059 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-run-openvswitch\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.286077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.286091 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-socket-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.286115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-sys-fs\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.286149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.286138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-sysconfig\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.286717 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.286178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vwmp\" (UniqueName: \"kubernetes.io/projected/d44ec06b-81b1-4bac-9abd-8efadd98b94a-kube-api-access-4vwmp\") pod \"iptables-alerter-m98lv\" (UID: \"d44ec06b-81b1-4bac-9abd-8efadd98b94a\") " pod="openshift-network-operator/iptables-alerter-m98lv" Apr 16 16:02:44.286717 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.286225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-run-netns\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.286717 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.286251 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-run\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.286717 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.286285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-var-lib-kubelet\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.286717 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.286301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrbw\" (UniqueName: \"kubernetes.io/projected/4c0f9a7c-7995-43e1-9141-56467d871fce-kube-api-access-rhrbw\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.286717 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.286606 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:44.286717 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.286679 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:02:44.287051 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.287038 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Apr 16 16:02:44.287111 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.287072 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gqfjh\"" Apr 16 16:02:44.287447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.287433 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Apr 16 16:02:44.312719 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.312676 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 15:57:43 +0000 UTC" deadline="2027-12-03 12:40:23.706364655 +0000 UTC" Apr 16 16:02:44.312719 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.312719 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14300h37m39.393650002s" Apr 16 16:02:44.375245 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.375218 2576 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" Apr 16 16:02:44.387170 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387139 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-device-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.387325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387180 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bb6679c-eb46-4326-bf8e-30829f808afc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.387325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-cni-dir\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.387325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-device-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.387325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387265 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-var-lib-kubelet\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.387494 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-conf-dir\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.387494 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387355 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-etc-openvswitch\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.387494 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387393 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.387494 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387413 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-etc-openvswitch\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.387494 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387421 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.387494 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387446 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-registration-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.387494 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387461 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.387494 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387474 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-sysctl-conf\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387497 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-systemd\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387517 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-registration-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387520 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-os-release\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387569 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-systemd\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-os-release\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387606 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-systemd-units\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387659 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-var-lib-openvswitch\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387682 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-systemd-units\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387682 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-sysctl-conf\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387708 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-var-lib-openvswitch\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387696 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-ovnkube-config\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387752 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c0f9a7c-7995-43e1-9141-56467d871fce-tmp\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387782 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a79498a-1173-46d9-921c-ebdf5b7ece1f-cni-binary-copy\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.387809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387809 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-slash\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387832 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-env-overrides\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387857 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-ovn-node-metrics-cert\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387881 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-kubernetes\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387906 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387914 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-slash\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387933 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-run-netns\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.387975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-var-lib-cni-multus\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388036 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-kubernetes\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388064 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-cni-bin\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388118 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-host\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388129 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388149 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-cnibin\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388164 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-host\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388122 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-cni-bin\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388173 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-var-lib-cni-bin\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388230 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5460383-84d2-4bd9-aa8b-82bbace94cb8-agent-certs\") pod \"konnectivity-agent-lg294\" (UID: \"b5460383-84d2-4bd9-aa8b-82bbace94cb8\") " pod="kube-system/konnectivity-agent-lg294" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388252 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-lib-modules\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.388559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388289 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-tuned\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388320 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388369 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d44ec06b-81b1-4bac-9abd-8efadd98b94a-iptables-alerter-script\") pod \"iptables-alerter-m98lv\" (UID: \"d44ec06b-81b1-4bac-9abd-8efadd98b94a\") " pod="openshift-network-operator/iptables-alerter-m98lv" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388393 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-ovnkube-config\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388478 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-lib-modules\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388488 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-run-openvswitch\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388396 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-run-openvswitch\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388528 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388555 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-sysconfig\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388582 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4vwmp\" (UniqueName: \"kubernetes.io/projected/d44ec06b-81b1-4bac-9abd-8efadd98b94a-kube-api-access-4vwmp\") pod \"iptables-alerter-m98lv\" (UID: \"d44ec06b-81b1-4bac-9abd-8efadd98b94a\") " pod="openshift-network-operator/iptables-alerter-m98lv" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388606 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-run-netns\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388492 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-env-overrides\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388633 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-cnibin\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388658 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0020ce4c-790d-41fd-9e3a-6ada38675cda-serviceca\") pod \"node-ca-bgdhp\" (UID: \"0020ce4c-790d-41fd-9e3a-6ada38675cda\") " pod="openshift-image-registry/node-ca-bgdhp" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-sysconfig\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388701 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-run-systemd\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388732 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-run-ovn-kubernetes\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.389358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.388747 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5460383-84d2-4bd9-aa8b-82bbace94cb8-konnectivity-ca\") pod \"konnectivity-agent-lg294\" (UID: \"b5460383-84d2-4bd9-aa8b-82bbace94cb8\") " pod="kube-system/konnectivity-agent-lg294" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389028 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-run-systemd\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389042 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-run-netns\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389065 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d44ec06b-81b1-4bac-9abd-8efadd98b94a-iptables-alerter-script\") pod \"iptables-alerter-m98lv\" (UID: \"d44ec06b-81b1-4bac-9abd-8efadd98b94a\") " pod="openshift-network-operator/iptables-alerter-m98lv" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389070 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-modprobe-d\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389126 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnkk4\" (UniqueName: \"kubernetes.io/projected/3a79498a-1173-46d9-921c-ebdf5b7ece1f-kube-api-access-nnkk4\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389160 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-kubelet\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389188 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-ovnkube-script-lib\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389205 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-kubelet\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389212 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-sysctl-d\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389164 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-modprobe-d\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bb6679c-eb46-4326-bf8e-30829f808afc-cni-binary-copy\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl899\" (UniqueName: \"kubernetes.io/projected/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-kube-api-access-gl899\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389297 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-daemon-config\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-run-multus-certs\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389346 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-node-log\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389374 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-csvfm\" (UniqueName: \"kubernetes.io/projected/0369ef86-452e-4148-ab71-6ccca93defc5-kube-api-access-csvfm\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.390149 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389402 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bfj\" (UniqueName: \"kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj\") pod \"network-check-target-2lh6g\" (UID: \"23dd67cf-2731-4685-80f7-f4a1e75b6476\") " pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389428 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-cni-netd\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389453 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nkk2q\" (UniqueName: \"kubernetes.io/projected/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-kube-api-access-nkk2q\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5zn8\" (UniqueName: \"kubernetes.io/projected/4bb6679c-eb46-4326-bf8e-30829f808afc-kube-api-access-z5zn8\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389537 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-log-socket\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389570 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0020ce4c-790d-41fd-9e3a-6ada38675cda-host\") pod \"node-ca-bgdhp\" (UID: \"0020ce4c-790d-41fd-9e3a-6ada38675cda\") " pod="openshift-image-registry/node-ca-bgdhp" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389597 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zslfc\" (UniqueName: \"kubernetes.io/projected/0020ce4c-790d-41fd-9e3a-6ada38675cda-kube-api-access-zslfc\") pod \"node-ca-bgdhp\" (UID: \"0020ce4c-790d-41fd-9e3a-6ada38675cda\") " pod="openshift-image-registry/node-ca-bgdhp" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389630 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-system-cni-dir\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389628 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b5460383-84d2-4bd9-aa8b-82bbace94cb8-konnectivity-ca\") pod \"konnectivity-agent-lg294\" (UID: \"b5460383-84d2-4bd9-aa8b-82bbace94cb8\") " pod="kube-system/konnectivity-agent-lg294" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389665 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389747 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-etc-selinux\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389752 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-ovnkube-script-lib\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389819 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-host-cni-netd\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389943 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-sysctl-d\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.389991 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-node-log\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390057 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-log-socket\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390142 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-sys\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.390912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390175 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-hostroot\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390212 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d44ec06b-81b1-4bac-9abd-8efadd98b94a-host-slash\") pod \"iptables-alerter-m98lv\" (UID: \"d44ec06b-81b1-4bac-9abd-8efadd98b94a\") " pod="openshift-network-operator/iptables-alerter-m98lv" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390232 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-sys\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390256 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-socket-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390285 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-sys-fs\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390299 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d44ec06b-81b1-4bac-9abd-8efadd98b94a-host-slash\") pod \"iptables-alerter-m98lv\" (UID: \"d44ec06b-81b1-4bac-9abd-8efadd98b94a\") " pod="openshift-network-operator/iptables-alerter-m98lv" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390310 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-var-lib-kubelet\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390378 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-sys-fs\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390391 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-var-lib-kubelet\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390413 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bb6679c-eb46-4326-bf8e-30829f808afc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0369ef86-452e-4148-ab71-6ccca93defc5-socket-dir\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390444 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-socket-dir-parent\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390473 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-run\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390500 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrbw\" (UniqueName: \"kubernetes.io/projected/4c0f9a7c-7995-43e1-9141-56467d871fce-kube-api-access-rhrbw\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-system-cni-dir\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390553 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-run-k8s-cni-cncf-io\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-etc-kubernetes\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.391663 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390617 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-run-ovn\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.392320 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390647 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4c0f9a7c-7995-43e1-9141-56467d871fce-run\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.392320 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.390699 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-run-ovn\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.392320 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.391615 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4c0f9a7c-7995-43e1-9141-56467d871fce-etc-tuned\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.392320 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.391656 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b5460383-84d2-4bd9-aa8b-82bbace94cb8-agent-certs\") pod \"konnectivity-agent-lg294\" (UID: \"b5460383-84d2-4bd9-aa8b-82bbace94cb8\") " pod="kube-system/konnectivity-agent-lg294" Apr 16 16:02:44.392320 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.391819 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-ovn-node-metrics-cert\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.392320 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.391861 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c0f9a7c-7995-43e1-9141-56467d871fce-tmp\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.399435 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.399410 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:02:44.399553 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.399439 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:02:44.399553 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.399453 2576 projected.go:194] Error preparing data for projected volume kube-api-access-q8bfj for pod openshift-network-diagnostics/network-check-target-2lh6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:02:44.399553 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.399519 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj podName:23dd67cf-2731-4685-80f7-f4a1e75b6476 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:44.899491795 +0000 UTC m=+3.158238583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q8bfj" (UniqueName: "kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj") pod "network-check-target-2lh6g" (UID: "23dd67cf-2731-4685-80f7-f4a1e75b6476") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:02:44.405118 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.405097 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkk2q\" (UniqueName: \"kubernetes.io/projected/c78e2321-0a40-4d26-89b3-afd2d6eee0a6-kube-api-access-nkk2q\") pod \"ovnkube-node-gtjzw\" (UID: \"c78e2321-0a40-4d26-89b3-afd2d6eee0a6\") " pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.405213 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.405178 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrbw\" (UniqueName: \"kubernetes.io/projected/4c0f9a7c-7995-43e1-9141-56467d871fce-kube-api-access-rhrbw\") pod \"tuned-q5jgq\" (UID: \"4c0f9a7c-7995-43e1-9141-56467d871fce\") " pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.405267 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.405221 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vwmp\" (UniqueName: \"kubernetes.io/projected/d44ec06b-81b1-4bac-9abd-8efadd98b94a-kube-api-access-4vwmp\") pod \"iptables-alerter-m98lv\" (UID: \"d44ec06b-81b1-4bac-9abd-8efadd98b94a\") " pod="openshift-network-operator/iptables-alerter-m98lv" Apr 16 16:02:44.405267 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.405253 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csvfm\" (UniqueName: \"kubernetes.io/projected/0369ef86-452e-4148-ab71-6ccca93defc5-kube-api-access-csvfm\") pod \"aws-ebs-csi-driver-node-8z2lq\" (UID: \"0369ef86-452e-4148-ab71-6ccca93defc5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.417840 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.417794 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" event={"ID":"a6c9c3cf395cb0ad946e6c0c60278a83","Type":"ContainerStarted","Data":"e81818b7d015c25dca06b802525ac118be9aa114bef8ad5adc11ef395ad7a322"} Apr 16 16:02:44.418882 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.418859 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" event={"ID":"2c91eb3ac6a0e55d0bbe3b604fc28906","Type":"ContainerStarted","Data":"a2b4f559cf45e920f7e224bed1482c64bb5092b40eb7e389581e07f783dd1f8a"} Apr 16 16:02:44.491566 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491533 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-os-release\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.491566 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491576 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a79498a-1173-46d9-921c-ebdf5b7ece1f-cni-binary-copy\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.491822 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491606 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.491822 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491651 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-run-netns\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.491822 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491695 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-var-lib-cni-multus\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.491822 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491697 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-os-release\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.491822 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491720 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-cnibin\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.491822 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491725 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-run-netns\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.491822 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491748 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-var-lib-cni-multus\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.491822 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491775 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-cnibin\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.491822 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491776 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-var-lib-cni-bin\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.491822 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491810 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-var-lib-cni-bin\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.491822 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491822 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491857 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-cnibin\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491882 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0020ce4c-790d-41fd-9e3a-6ada38675cda-serviceca\") pod \"node-ca-bgdhp\" (UID: \"0020ce4c-790d-41fd-9e3a-6ada38675cda\") " pod="openshift-image-registry/node-ca-bgdhp" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.491899 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491907 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nnkk4\" (UniqueName: \"kubernetes.io/projected/3a79498a-1173-46d9-921c-ebdf5b7ece1f-kube-api-access-nnkk4\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491933 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bb6679c-eb46-4326-bf8e-30829f808afc-cni-binary-copy\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491960 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gl899\" (UniqueName: \"kubernetes.io/projected/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-kube-api-access-gl899\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491973 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-cnibin\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.491988 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-daemon-config\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492039 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-run-multus-certs\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.492078 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs podName:cd5f8939-2604-4ca4-8d1e-05dc7f86fa41 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:44.992057846 +0000 UTC m=+3.250804667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs") pod "network-metrics-daemon-fpztb" (UID: "cd5f8939-2604-4ca4-8d1e-05dc7f86fa41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492086 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-run-multus-certs\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492117 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z5zn8\" (UniqueName: \"kubernetes.io/projected/4bb6679c-eb46-4326-bf8e-30829f808afc-kube-api-access-z5zn8\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492145 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0020ce4c-790d-41fd-9e3a-6ada38675cda-host\") pod \"node-ca-bgdhp\" (UID: \"0020ce4c-790d-41fd-9e3a-6ada38675cda\") " pod="openshift-image-registry/node-ca-bgdhp" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492171 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zslfc\" (UniqueName: \"kubernetes.io/projected/0020ce4c-790d-41fd-9e3a-6ada38675cda-kube-api-access-zslfc\") pod \"node-ca-bgdhp\" (UID: \"0020ce4c-790d-41fd-9e3a-6ada38675cda\") " pod="openshift-image-registry/node-ca-bgdhp" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492218 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-system-cni-dir\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492242 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-hostroot\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.492930 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492275 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bb6679c-eb46-4326-bf8e-30829f808afc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492314 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-socket-dir-parent\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492350 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-system-cni-dir\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492377 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-run-k8s-cni-cncf-io\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492396 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-system-cni-dir\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492407 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-etc-kubernetes\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492424 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0020ce4c-790d-41fd-9e3a-6ada38675cda-serviceca\") pod \"node-ca-bgdhp\" (UID: \"0020ce4c-790d-41fd-9e3a-6ada38675cda\") " pod="openshift-image-registry/node-ca-bgdhp" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492437 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bb6679c-eb46-4326-bf8e-30829f808afc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492424 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-socket-dir-parent\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492458 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-hostroot\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492489 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-cni-dir\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492516 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-run-k8s-cni-cncf-io\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492525 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-var-lib-kubelet\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492558 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-system-cni-dir\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492544 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-daemon-config\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492553 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a79498a-1173-46d9-921c-ebdf5b7ece1f-cni-binary-copy\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492440 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0020ce4c-790d-41fd-9e3a-6ada38675cda-host\") pod \"node-ca-bgdhp\" (UID: \"0020ce4c-790d-41fd-9e3a-6ada38675cda\") " pod="openshift-image-registry/node-ca-bgdhp" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492562 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-conf-dir\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.493729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492348 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.494600 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492612 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4bb6679c-eb46-4326-bf8e-30829f808afc-cni-binary-copy\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.494600 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492624 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-conf-dir\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.494600 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492636 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-multus-cni-dir\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.494600 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492636 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-host-var-lib-kubelet\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.494600 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492644 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a79498a-1173-46d9-921c-ebdf5b7ece1f-etc-kubernetes\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.494600 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492685 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-os-release\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.494600 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492748 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4bb6679c-eb46-4326-bf8e-30829f808afc-os-release\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.494600 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.492866 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4bb6679c-eb46-4326-bf8e-30829f808afc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.494600 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.493377 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4bb6679c-eb46-4326-bf8e-30829f808afc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.501470 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.501444 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl899\" (UniqueName: \"kubernetes.io/projected/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-kube-api-access-gl899\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:44.501650 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.501494 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zslfc\" (UniqueName: \"kubernetes.io/projected/0020ce4c-790d-41fd-9e3a-6ada38675cda-kube-api-access-zslfc\") pod \"node-ca-bgdhp\" (UID: \"0020ce4c-790d-41fd-9e3a-6ada38675cda\") " pod="openshift-image-registry/node-ca-bgdhp" Apr 16 16:02:44.501650 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.501607 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5zn8\" (UniqueName: \"kubernetes.io/projected/4bb6679c-eb46-4326-bf8e-30829f808afc-kube-api-access-z5zn8\") pod \"multus-additional-cni-plugins-dfnlw\" (UID: \"4bb6679c-eb46-4326-bf8e-30829f808afc\") " pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.502604 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.502586 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnkk4\" (UniqueName: \"kubernetes.io/projected/3a79498a-1173-46d9-921c-ebdf5b7ece1f-kube-api-access-nnkk4\") pod \"multus-68jlg\" (UID: \"3a79498a-1173-46d9-921c-ebdf5b7ece1f\") " pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.580719 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.580689 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" Apr 16 16:02:44.591297 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.591271 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m98lv" Apr 16 16:02:44.597046 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.597025 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:02:44.601634 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.601616 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lg294" Apr 16 16:02:44.609146 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.609125 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" Apr 16 16:02:44.614693 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.614671 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bgdhp" Apr 16 16:02:44.623285 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.623268 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-68jlg" Apr 16 16:02:44.626766 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.626742 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" Apr 16 16:02:44.996147 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.996117 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:44.996335 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:44.996164 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bfj\" (UniqueName: \"kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj\") pod \"network-check-target-2lh6g\" (UID: \"23dd67cf-2731-4685-80f7-f4a1e75b6476\") " pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:44.996335 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.996269 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:02:44.996335 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.996284 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:02:44.996335 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.996301 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:02:44.996335 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.996313 2576 projected.go:194] Error preparing data for projected volume kube-api-access-q8bfj for pod openshift-network-diagnostics/network-check-target-2lh6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:02:44.996335 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.996334 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs podName:cd5f8939-2604-4ca4-8d1e-05dc7f86fa41 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:45.996316392 +0000 UTC m=+4.255063200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs") pod "network-metrics-daemon-fpztb" (UID: "cd5f8939-2604-4ca4-8d1e-05dc7f86fa41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:02:44.996599 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:44.996362 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj podName:23dd67cf-2731-4685-80f7-f4a1e75b6476 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:45.996345257 +0000 UTC m=+4.255092059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q8bfj" (UniqueName: "kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj") pod "network-check-target-2lh6g" (UID: "23dd67cf-2731-4685-80f7-f4a1e75b6476") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:02:45.123280 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:45.123065 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5460383_84d2_4bd9_aa8b_82bbace94cb8.slice/crio-c8aa1edcb29f0b5b61f17dcbf5661a11e07fb9350a6c173934f06c50e3085571 WatchSource:0}: Error finding container c8aa1edcb29f0b5b61f17dcbf5661a11e07fb9350a6c173934f06c50e3085571: Status 404 returned error can't find the container with id c8aa1edcb29f0b5b61f17dcbf5661a11e07fb9350a6c173934f06c50e3085571 Apr 16 16:02:45.124684 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:45.124661 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0369ef86_452e_4148_ab71_6ccca93defc5.slice/crio-e243d2979f750b087fc70fc5514fe4dc66f94e47e92cee17d8df493dbb0137cc WatchSource:0}: Error finding container e243d2979f750b087fc70fc5514fe4dc66f94e47e92cee17d8df493dbb0137cc: Status 404 returned error can't find the container with id e243d2979f750b087fc70fc5514fe4dc66f94e47e92cee17d8df493dbb0137cc Apr 16 16:02:45.127707 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:45.127606 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44ec06b_81b1_4bac_9abd_8efadd98b94a.slice/crio-c5cbf90936feb995e0b2e3af491244a65a9ab102bf5c5b25f0e00ae481a277dc WatchSource:0}: Error finding container c5cbf90936feb995e0b2e3af491244a65a9ab102bf5c5b25f0e00ae481a277dc: Status 404 returned error can't find the container with id c5cbf90936feb995e0b2e3af491244a65a9ab102bf5c5b25f0e00ae481a277dc Apr 16 16:02:45.128370 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:45.128343 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a79498a_1173_46d9_921c_ebdf5b7ece1f.slice/crio-761311d9b84b6e065db4228e57b24e5c4e400d633354382002337389dbb67231 WatchSource:0}: Error finding container 761311d9b84b6e065db4228e57b24e5c4e400d633354382002337389dbb67231: Status 404 returned error can't find the container with id 761311d9b84b6e065db4228e57b24e5c4e400d633354382002337389dbb67231 Apr 16 16:02:45.129205 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:45.129176 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc78e2321_0a40_4d26_89b3_afd2d6eee0a6.slice/crio-fc4e629420c781ac38d56727dda4a26634222c23dce721bfed4b09b1ff3df395 WatchSource:0}: Error finding container fc4e629420c781ac38d56727dda4a26634222c23dce721bfed4b09b1ff3df395: Status 404 returned error can't find the container with id fc4e629420c781ac38d56727dda4a26634222c23dce721bfed4b09b1ff3df395 Apr 16 16:02:45.130686 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:45.130591 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0020ce4c_790d_41fd_9e3a_6ada38675cda.slice/crio-f441a8034024e878933dcf4a3f1233fbbe0cc644976969efeded5719f4294295 WatchSource:0}: Error finding container f441a8034024e878933dcf4a3f1233fbbe0cc644976969efeded5719f4294295: Status 404 returned error can't find the container with id f441a8034024e878933dcf4a3f1233fbbe0cc644976969efeded5719f4294295 Apr 16 16:02:45.131739 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:45.131574 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb6679c_eb46_4326_bf8e_30829f808afc.slice/crio-d36aca5ae17ea5a0380bbaa79ceb450e64d6c384801873a57d4fe862326e8f24 WatchSource:0}: Error finding container d36aca5ae17ea5a0380bbaa79ceb450e64d6c384801873a57d4fe862326e8f24: Status 404 returned error can't find the container with id d36aca5ae17ea5a0380bbaa79ceb450e64d6c384801873a57d4fe862326e8f24 Apr 16 16:02:45.132959 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:45.132940 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0f9a7c_7995_43e1_9141_56467d871fce.slice/crio-b2c38741343d0d6f78b37a069a42d01c90dccbfe85abf9c128f660966028feaf WatchSource:0}: Error finding container b2c38741343d0d6f78b37a069a42d01c90dccbfe85abf9c128f660966028feaf: Status 404 returned error can't find the container with id b2c38741343d0d6f78b37a069a42d01c90dccbfe85abf9c128f660966028feaf Apr 16 16:02:45.313708 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.313614 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-04-15 15:57:43 +0000 UTC" deadline="2027-09-22 17:13:22.113400786 +0000 UTC" Apr 16 16:02:45.313708 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.313646 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12577h10m36.799757226s" Apr 16 16:02:45.421755 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.421716 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m98lv" event={"ID":"d44ec06b-81b1-4bac-9abd-8efadd98b94a","Type":"ContainerStarted","Data":"c5cbf90936feb995e0b2e3af491244a65a9ab102bf5c5b25f0e00ae481a277dc"} Apr 16 16:02:45.423847 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.423744 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" event={"ID":"0369ef86-452e-4148-ab71-6ccca93defc5","Type":"ContainerStarted","Data":"e243d2979f750b087fc70fc5514fe4dc66f94e47e92cee17d8df493dbb0137cc"} Apr 16 16:02:45.425339 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.425313 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lg294" event={"ID":"b5460383-84d2-4bd9-aa8b-82bbace94cb8","Type":"ContainerStarted","Data":"c8aa1edcb29f0b5b61f17dcbf5661a11e07fb9350a6c173934f06c50e3085571"} Apr 16 16:02:45.427856 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.427819 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" event={"ID":"a6c9c3cf395cb0ad946e6c0c60278a83","Type":"ContainerStarted","Data":"50fa1671b0fadea5cf79ba74179f40d4d5f010dabbd88120db5c33ff17d906b3"} Apr 16 16:02:45.428850 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.428816 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" event={"ID":"4c0f9a7c-7995-43e1-9141-56467d871fce","Type":"ContainerStarted","Data":"b2c38741343d0d6f78b37a069a42d01c90dccbfe85abf9c128f660966028feaf"} Apr 16 16:02:45.430058 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.430033 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" event={"ID":"4bb6679c-eb46-4326-bf8e-30829f808afc","Type":"ContainerStarted","Data":"d36aca5ae17ea5a0380bbaa79ceb450e64d6c384801873a57d4fe862326e8f24"} Apr 16 16:02:45.431289 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.431178 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bgdhp" event={"ID":"0020ce4c-790d-41fd-9e3a-6ada38675cda","Type":"ContainerStarted","Data":"f441a8034024e878933dcf4a3f1233fbbe0cc644976969efeded5719f4294295"} Apr 16 16:02:45.432208 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.432188 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" event={"ID":"c78e2321-0a40-4d26-89b3-afd2d6eee0a6","Type":"ContainerStarted","Data":"fc4e629420c781ac38d56727dda4a26634222c23dce721bfed4b09b1ff3df395"} Apr 16 16:02:45.433325 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.433303 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68jlg" event={"ID":"3a79498a-1173-46d9-921c-ebdf5b7ece1f","Type":"ContainerStarted","Data":"761311d9b84b6e065db4228e57b24e5c4e400d633354382002337389dbb67231"} Apr 16 16:02:45.441169 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:45.441121 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-177.ec2.internal" podStartSLOduration=2.441109409 podStartE2EDuration="2.441109409s" podCreationTimestamp="2026-04-16 16:02:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:02:45.440694305 +0000 UTC m=+3.699441115" watchObservedRunningTime="2026-04-16 16:02:45.441109409 +0000 UTC m=+3.699856219" Apr 16 16:02:46.004927 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.004884 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bfj\" (UniqueName: \"kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj\") pod \"network-check-target-2lh6g\" (UID: \"23dd67cf-2731-4685-80f7-f4a1e75b6476\") " pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:46.005079 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.004974 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:46.005145 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:46.005126 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:02:46.005196 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:46.005192 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs podName:cd5f8939-2604-4ca4-8d1e-05dc7f86fa41 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:48.005172886 +0000 UTC m=+6.263919700 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs") pod "network-metrics-daemon-fpztb" (UID: "cd5f8939-2604-4ca4-8d1e-05dc7f86fa41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:02:46.005614 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:46.005593 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:02:46.005712 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:46.005619 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:02:46.005712 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:46.005632 2576 projected.go:194] Error preparing data for projected volume kube-api-access-q8bfj for pod openshift-network-diagnostics/network-check-target-2lh6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:02:46.005712 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:46.005679 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj podName:23dd67cf-2731-4685-80f7-f4a1e75b6476 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:48.005664395 +0000 UTC m=+6.264411186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q8bfj" (UniqueName: "kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj") pod "network-check-target-2lh6g" (UID: "23dd67cf-2731-4685-80f7-f4a1e75b6476") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:02:46.021604 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.021558 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-klzvx"] Apr 16 16:02:46.024578 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.024554 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klzvx" Apr 16 16:02:46.032828 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.032801 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-279dv\"" Apr 16 16:02:46.032944 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.032925 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Apr 16 16:02:46.033069 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.033053 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Apr 16 16:02:46.106271 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.106235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd45d1df-4307-46b1-878a-6073d5e56455-hosts-file\") pod \"node-resolver-klzvx\" (UID: \"dd45d1df-4307-46b1-878a-6073d5e56455\") " pod="openshift-dns/node-resolver-klzvx" Apr 16 16:02:46.106396 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.106287 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd45d1df-4307-46b1-878a-6073d5e56455-tmp-dir\") pod \"node-resolver-klzvx\" (UID: \"dd45d1df-4307-46b1-878a-6073d5e56455\") " pod="openshift-dns/node-resolver-klzvx" Apr 16 16:02:46.106396 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.106364 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rtr\" (UniqueName: \"kubernetes.io/projected/dd45d1df-4307-46b1-878a-6073d5e56455-kube-api-access-r9rtr\") pod \"node-resolver-klzvx\" (UID: \"dd45d1df-4307-46b1-878a-6073d5e56455\") " pod="openshift-dns/node-resolver-klzvx" Apr 16 16:02:46.206907 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.206826 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rtr\" (UniqueName: \"kubernetes.io/projected/dd45d1df-4307-46b1-878a-6073d5e56455-kube-api-access-r9rtr\") pod \"node-resolver-klzvx\" (UID: \"dd45d1df-4307-46b1-878a-6073d5e56455\") " pod="openshift-dns/node-resolver-klzvx" Apr 16 16:02:46.206907 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.206900 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd45d1df-4307-46b1-878a-6073d5e56455-hosts-file\") pod \"node-resolver-klzvx\" (UID: \"dd45d1df-4307-46b1-878a-6073d5e56455\") " pod="openshift-dns/node-resolver-klzvx" Apr 16 16:02:46.207144 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.206930 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd45d1df-4307-46b1-878a-6073d5e56455-tmp-dir\") pod \"node-resolver-klzvx\" (UID: \"dd45d1df-4307-46b1-878a-6073d5e56455\") " pod="openshift-dns/node-resolver-klzvx" Apr 16 16:02:46.207328 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.207307 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dd45d1df-4307-46b1-878a-6073d5e56455-tmp-dir\") pod \"node-resolver-klzvx\" (UID: \"dd45d1df-4307-46b1-878a-6073d5e56455\") " pod="openshift-dns/node-resolver-klzvx" Apr 16 16:02:46.207715 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.207698 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dd45d1df-4307-46b1-878a-6073d5e56455-hosts-file\") pod \"node-resolver-klzvx\" (UID: \"dd45d1df-4307-46b1-878a-6073d5e56455\") " pod="openshift-dns/node-resolver-klzvx" Apr 16 16:02:46.237270 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.237237 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rtr\" (UniqueName: \"kubernetes.io/projected/dd45d1df-4307-46b1-878a-6073d5e56455-kube-api-access-r9rtr\") pod \"node-resolver-klzvx\" (UID: \"dd45d1df-4307-46b1-878a-6073d5e56455\") " pod="openshift-dns/node-resolver-klzvx" Apr 16 16:02:46.335807 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.335678 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-klzvx" Apr 16 16:02:46.358006 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:02:46.353900 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd45d1df_4307_46b1_878a_6073d5e56455.slice/crio-e766c08420a925279d40ed5dbebd944cd02a75c9464cdfab94d0efc3339cf338 WatchSource:0}: Error finding container e766c08420a925279d40ed5dbebd944cd02a75c9464cdfab94d0efc3339cf338: Status 404 returned error can't find the container with id e766c08420a925279d40ed5dbebd944cd02a75c9464cdfab94d0efc3339cf338 Apr 16 16:02:46.415199 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.415162 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:46.415351 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:46.415291 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:02:46.415832 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.415633 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:46.415832 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:46.415760 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:02:46.437509 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.437473 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klzvx" event={"ID":"dd45d1df-4307-46b1-878a-6073d5e56455","Type":"ContainerStarted","Data":"e766c08420a925279d40ed5dbebd944cd02a75c9464cdfab94d0efc3339cf338"} Apr 16 16:02:46.443533 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.443506 2576 generic.go:358] "Generic (PLEG): container finished" podID="2c91eb3ac6a0e55d0bbe3b604fc28906" containerID="065fe126eadff031b825dcc85c98a13fd93af0e80e2819d6c1f85ee95b0c4384" exitCode=0 Apr 16 16:02:46.444520 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:46.444494 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" event={"ID":"2c91eb3ac6a0e55d0bbe3b604fc28906","Type":"ContainerDied","Data":"065fe126eadff031b825dcc85c98a13fd93af0e80e2819d6c1f85ee95b0c4384"} Apr 16 16:02:47.462570 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:47.462500 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" event={"ID":"2c91eb3ac6a0e55d0bbe3b604fc28906","Type":"ContainerStarted","Data":"5d51a617c04b136d6201db5a4671084f53243641f9efa91b91fd3293a0701718"} Apr 16 16:02:47.476046 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:47.475973 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-177.ec2.internal" podStartSLOduration=4.475954618 podStartE2EDuration="4.475954618s" podCreationTimestamp="2026-04-16 16:02:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:02:47.475571715 +0000 UTC m=+5.734318527" watchObservedRunningTime="2026-04-16 16:02:47.475954618 +0000 UTC m=+5.734701429" Apr 16 16:02:48.024265 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:48.024199 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:48.024265 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:48.024262 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bfj\" (UniqueName: \"kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj\") pod \"network-check-target-2lh6g\" (UID: \"23dd67cf-2731-4685-80f7-f4a1e75b6476\") " pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:48.024582 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:48.024413 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:02:48.024582 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:48.024434 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:02:48.024582 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:48.024447 2576 projected.go:194] Error preparing data for projected volume kube-api-access-q8bfj for pod openshift-network-diagnostics/network-check-target-2lh6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:02:48.024582 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:48.024508 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj podName:23dd67cf-2731-4685-80f7-f4a1e75b6476 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:52.024487914 +0000 UTC m=+10.283234708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q8bfj" (UniqueName: "kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj") pod "network-check-target-2lh6g" (UID: "23dd67cf-2731-4685-80f7-f4a1e75b6476") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:02:48.024956 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:48.024918 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:02:48.025098 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:48.024972 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs podName:cd5f8939-2604-4ca4-8d1e-05dc7f86fa41 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:52.024956951 +0000 UTC m=+10.283703748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs") pod "network-metrics-daemon-fpztb" (UID: "cd5f8939-2604-4ca4-8d1e-05dc7f86fa41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:02:48.416872 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:48.415415 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:48.416872 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:48.415557 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:02:48.416872 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:48.416053 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:48.416872 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:48.416150 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:02:50.414635 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:50.414591 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:50.415055 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:50.414720 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:02:50.415055 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:50.414597 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:50.415055 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:50.414832 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:02:52.056678 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:52.055861 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:52.056678 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:52.055921 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bfj\" (UniqueName: \"kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj\") pod \"network-check-target-2lh6g\" (UID: \"23dd67cf-2731-4685-80f7-f4a1e75b6476\") " pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:52.056678 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:52.056094 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:02:52.056678 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:52.056113 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:02:52.056678 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:52.056126 2576 projected.go:194] Error preparing data for projected volume kube-api-access-q8bfj for pod openshift-network-diagnostics/network-check-target-2lh6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:02:52.056678 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:52.056185 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj podName:23dd67cf-2731-4685-80f7-f4a1e75b6476 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:00.056168447 +0000 UTC m=+18.314915240 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q8bfj" (UniqueName: "kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj") pod "network-check-target-2lh6g" (UID: "23dd67cf-2731-4685-80f7-f4a1e75b6476") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:02:52.056678 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:52.056598 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:02:52.056678 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:52.056645 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs podName:cd5f8939-2604-4ca4-8d1e-05dc7f86fa41 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:00.056630238 +0000 UTC m=+18.315377033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs") pod "network-metrics-daemon-fpztb" (UID: "cd5f8939-2604-4ca4-8d1e-05dc7f86fa41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:02:52.416421 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:52.415642 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:52.416421 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:52.415750 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:02:52.416421 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:52.416168 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:52.416421 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:52.416310 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:02:54.413999 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:54.413914 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:54.414448 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:54.413914 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:54.414448 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:54.414058 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:02:54.414448 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:54.414134 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:02:56.414749 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:56.414711 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:56.415205 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:56.414759 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:56.415205 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:56.414861 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:02:56.415205 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:56.414993 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:02:57.506662 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:57.506633 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cpt7l"] Apr 16 16:02:57.509438 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:57.509418 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:57.509547 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:57.509482 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpt7l" podUID="9b57199f-943e-4015-b87b-a582c2c06494" Apr 16 16:02:57.594529 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:57.594494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:57.594529 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:57.594538 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b57199f-943e-4015-b87b-a582c2c06494-kubelet-config\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:57.594711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:57.594559 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b57199f-943e-4015-b87b-a582c2c06494-dbus\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:57.695357 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:57.695321 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:57.695545 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:57.695369 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b57199f-943e-4015-b87b-a582c2c06494-kubelet-config\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:57.695545 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:57.695447 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9b57199f-943e-4015-b87b-a582c2c06494-kubelet-config\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:57.695545 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:57.695502 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:02:57.695680 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:57.695542 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b57199f-943e-4015-b87b-a582c2c06494-dbus\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:57.695680 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:57.695568 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret podName:9b57199f-943e-4015-b87b-a582c2c06494 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:58.195550568 +0000 UTC m=+16.454297376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret") pod "global-pull-secret-syncer-cpt7l" (UID: "9b57199f-943e-4015-b87b-a582c2c06494") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:02:57.695680 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:57.695675 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9b57199f-943e-4015-b87b-a582c2c06494-dbus\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:58.201245 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:58.201210 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:58.201419 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:58.201378 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:02:58.201477 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:58.201456 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret podName:9b57199f-943e-4015-b87b-a582c2c06494 nodeName:}" failed. No retries permitted until 2026-04-16 16:02:59.201434356 +0000 UTC m=+17.460181168 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret") pod "global-pull-secret-syncer-cpt7l" (UID: "9b57199f-943e-4015-b87b-a582c2c06494") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:02:58.414438 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:58.414367 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:02:58.414438 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:58.414390 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:02:58.414643 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:58.414503 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:02:58.414643 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:58.414607 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:02:59.209634 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:59.209551 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:59.210223 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:59.209729 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:02:59.210223 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:59.209815 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret podName:9b57199f-943e-4015-b87b-a582c2c06494 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:01.209795069 +0000 UTC m=+19.468541885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret") pod "global-pull-secret-syncer-cpt7l" (UID: "9b57199f-943e-4015-b87b-a582c2c06494") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:02:59.414729 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:02:59.414700 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:02:59.414904 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:02:59.414824 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpt7l" podUID="9b57199f-943e-4015-b87b-a582c2c06494" Apr 16 16:03:00.116123 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:00.116081 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:00.116319 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:00.116149 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bfj\" (UniqueName: \"kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj\") pod \"network-check-target-2lh6g\" (UID: \"23dd67cf-2731-4685-80f7-f4a1e75b6476\") " pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:00.116319 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:00.116243 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:03:00.116319 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:00.116276 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:03:00.116319 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:00.116289 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:03:00.116319 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:00.116299 2576 projected.go:194] Error preparing data for projected volume kube-api-access-q8bfj for pod openshift-network-diagnostics/network-check-target-2lh6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:03:00.116319 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:00.116315 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs podName:cd5f8939-2604-4ca4-8d1e-05dc7f86fa41 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:16.116299725 +0000 UTC m=+34.375046513 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs") pod "network-metrics-daemon-fpztb" (UID: "cd5f8939-2604-4ca4-8d1e-05dc7f86fa41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:03:00.116568 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:00.116343 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj podName:23dd67cf-2731-4685-80f7-f4a1e75b6476 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:16.116328523 +0000 UTC m=+34.375075345 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q8bfj" (UniqueName: "kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj") pod "network-check-target-2lh6g" (UID: "23dd67cf-2731-4685-80f7-f4a1e75b6476") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:03:00.414820 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:00.414739 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:00.414820 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:00.414782 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:00.415294 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:00.414881 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:03:00.415294 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:00.415008 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:03:01.225127 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:01.225081 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:01.225319 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:01.225268 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:03:01.225373 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:01.225357 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret podName:9b57199f-943e-4015-b87b-a582c2c06494 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:05.225333998 +0000 UTC m=+23.484080795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret") pod "global-pull-secret-syncer-cpt7l" (UID: "9b57199f-943e-4015-b87b-a582c2c06494") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:03:01.414086 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:01.414049 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:01.414274 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:01.414188 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpt7l" podUID="9b57199f-943e-4015-b87b-a582c2c06494" Apr 16 16:03:02.372905 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:02.372498 2576 cadvisor_stats_provider.go:525] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a79498a_1173_46d9_921c_ebdf5b7ece1f.slice/crio-6af424bf77c8f9d4accfe0d7e06692c081bf4c7e221a073d410a5282dba9638b.scope\": RecentStats: unable to find data in memory cache]" Apr 16 16:03:02.417120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.415944 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:02.417120 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:02.416221 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:03:02.417120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.416261 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:02.417120 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:02.416359 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:03:02.489248 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.489213 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-klzvx" event={"ID":"dd45d1df-4307-46b1-878a-6073d5e56455","Type":"ContainerStarted","Data":"b567daae6dee7584e02c898b842e5bd360dae687e8dafdef761e68b42b0a59ef"} Apr 16 16:03:02.491805 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.491779 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lg294" event={"ID":"b5460383-84d2-4bd9-aa8b-82bbace94cb8","Type":"ContainerStarted","Data":"2d3c70247a5993c6ff666fe3a247850df0158d3456b1c8f82b25910e0695ca1d"} Apr 16 16:03:02.493186 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.493167 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" event={"ID":"4c0f9a7c-7995-43e1-9141-56467d871fce","Type":"ContainerStarted","Data":"867ce7847b64cbc03a16ffa352c02d90a300bd9d8dc52183eb5c9473cfbb9031"} Apr 16 16:03:02.494532 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.494506 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" event={"ID":"4bb6679c-eb46-4326-bf8e-30829f808afc","Type":"ContainerStarted","Data":"b84e645187724c3b6ced02574553b64cbd54d5442cccbc2c0222943538c7ccd1"} Apr 16 16:03:02.495732 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.495713 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bgdhp" event={"ID":"0020ce4c-790d-41fd-9e3a-6ada38675cda","Type":"ContainerStarted","Data":"3416faab0038380a84ac230d1614e9d73bf2b1ac77110c87afd4d20573594506"} Apr 16 16:03:02.496937 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.496915 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" event={"ID":"c78e2321-0a40-4d26-89b3-afd2d6eee0a6","Type":"ContainerStarted","Data":"ec6cdceadab964783842e9e33bdda2af22c24e15823292f27ec549e59232b867"} Apr 16 16:03:02.498191 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.498170 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-68jlg" event={"ID":"3a79498a-1173-46d9-921c-ebdf5b7ece1f","Type":"ContainerStarted","Data":"6af424bf77c8f9d4accfe0d7e06692c081bf4c7e221a073d410a5282dba9638b"} Apr 16 16:03:02.518115 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.518073 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-klzvx" podStartSLOduration=0.732290972 podStartE2EDuration="16.518060232s" podCreationTimestamp="2026-04-16 16:02:46 +0000 UTC" firstStartedPulling="2026-04-16 16:02:46.357001751 +0000 UTC m=+4.615748551" lastFinishedPulling="2026-04-16 16:03:02.14277102 +0000 UTC m=+20.401517811" observedRunningTime="2026-04-16 16:03:02.505199529 +0000 UTC m=+20.763946339" watchObservedRunningTime="2026-04-16 16:03:02.518060232 +0000 UTC m=+20.776807044" Apr 16 16:03:02.518317 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.518291 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bgdhp" podStartSLOduration=11.54281137 podStartE2EDuration="20.518283856s" podCreationTimestamp="2026-04-16 16:02:42 +0000 UTC" firstStartedPulling="2026-04-16 16:02:45.133526612 +0000 UTC m=+3.392273404" lastFinishedPulling="2026-04-16 16:02:54.108999089 +0000 UTC m=+12.367745890" observedRunningTime="2026-04-16 16:03:02.517948805 +0000 UTC m=+20.776695619" watchObservedRunningTime="2026-04-16 16:03:02.518283856 +0000 UTC m=+20.777030665" Apr 16 16:03:02.553504 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.553468 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-68jlg" podStartSLOduration=3.5044150370000002 podStartE2EDuration="20.55345582s" podCreationTimestamp="2026-04-16 16:02:42 +0000 UTC" firstStartedPulling="2026-04-16 16:02:45.130813596 +0000 UTC m=+3.389560384" lastFinishedPulling="2026-04-16 16:03:02.179854364 +0000 UTC m=+20.438601167" observedRunningTime="2026-04-16 16:03:02.55306222 +0000 UTC m=+20.811809030" watchObservedRunningTime="2026-04-16 16:03:02.55345582 +0000 UTC m=+20.812202629" Apr 16 16:03:02.569970 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.569939 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q5jgq" podStartSLOduration=3.53574244 podStartE2EDuration="20.569927528s" podCreationTimestamp="2026-04-16 16:02:42 +0000 UTC" firstStartedPulling="2026-04-16 16:02:45.135346782 +0000 UTC m=+3.394093570" lastFinishedPulling="2026-04-16 16:03:02.169531866 +0000 UTC m=+20.428278658" observedRunningTime="2026-04-16 16:03:02.569883276 +0000 UTC m=+20.828630087" watchObservedRunningTime="2026-04-16 16:03:02.569927528 +0000 UTC m=+20.828674330" Apr 16 16:03:02.583808 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.583775 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lg294" podStartSLOduration=3.584607058 podStartE2EDuration="20.583762767s" podCreationTimestamp="2026-04-16 16:02:42 +0000 UTC" firstStartedPulling="2026-04-16 16:02:45.125722804 +0000 UTC m=+3.384469606" lastFinishedPulling="2026-04-16 16:03:02.124878523 +0000 UTC m=+20.383625315" observedRunningTime="2026-04-16 16:03:02.583469483 +0000 UTC m=+20.842216293" watchObservedRunningTime="2026-04-16 16:03:02.583762767 +0000 UTC m=+20.842509610" Apr 16 16:03:02.773076 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.772845 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lg294" Apr 16 16:03:02.773433 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:02.773417 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lg294" Apr 16 16:03:03.414666 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.414637 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:03.415128 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:03.414733 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpt7l" podUID="9b57199f-943e-4015-b87b-a582c2c06494" Apr 16 16:03:03.500984 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.500955 2576 generic.go:358] "Generic (PLEG): container finished" podID="4bb6679c-eb46-4326-bf8e-30829f808afc" containerID="b84e645187724c3b6ced02574553b64cbd54d5442cccbc2c0222943538c7ccd1" exitCode=0 Apr 16 16:03:03.501124 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.501046 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" event={"ID":"4bb6679c-eb46-4326-bf8e-30829f808afc","Type":"ContainerDied","Data":"b84e645187724c3b6ced02574553b64cbd54d5442cccbc2c0222943538c7ccd1"} Apr 16 16:03:03.503594 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.503575 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:03:03.504303 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.504124 2576 generic.go:358] "Generic (PLEG): container finished" podID="c78e2321-0a40-4d26-89b3-afd2d6eee0a6" containerID="ce18aa0e88969c3fd61d2a327769cb9e83ce1431744ac2419a98ba4f80783dd9" exitCode=1 Apr 16 16:03:03.504303 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.504200 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" event={"ID":"c78e2321-0a40-4d26-89b3-afd2d6eee0a6","Type":"ContainerStarted","Data":"628e86ed1ea50fd982e33e1a97f205ba31db31b4f4521f7f37f43a03394edc37"} Apr 16 16:03:03.504303 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.504222 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" event={"ID":"c78e2321-0a40-4d26-89b3-afd2d6eee0a6","Type":"ContainerStarted","Data":"acd5b100545f48762c60511760882ae8e58662b193b9f4dcbdc9083061a04713"} Apr 16 16:03:03.504303 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.504234 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" event={"ID":"c78e2321-0a40-4d26-89b3-afd2d6eee0a6","Type":"ContainerStarted","Data":"34953921b619b1cb6ffaa5d880e19877494ed7892074e7ce1191b4aa5ca1bbe1"} Apr 16 16:03:03.504303 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.504248 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" event={"ID":"c78e2321-0a40-4d26-89b3-afd2d6eee0a6","Type":"ContainerStarted","Data":"7fb9784d84e6b3da9a89cfb375e179351fce8c339eb11c71aa8ea51f3f34862f"} Apr 16 16:03:03.504303 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.504261 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" event={"ID":"c78e2321-0a40-4d26-89b3-afd2d6eee0a6","Type":"ContainerDied","Data":"ce18aa0e88969c3fd61d2a327769cb9e83ce1431744ac2419a98ba4f80783dd9"} Apr 16 16:03:03.506252 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.506230 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" event={"ID":"0369ef86-452e-4148-ab71-6ccca93defc5","Type":"ContainerStarted","Data":"4bcfd39637c48894943071f048c30592e517f0db04f3f225aaa0011ee0b1ff24"} Apr 16 16:03:03.904226 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:03.904034 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Apr 16 16:03:04.341328 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:04.341226 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-04-16T16:03:03.904221786Z","UUID":"db26dac6-06f1-49af-9618-3200a0a65552","Handler":null,"Name":"","Endpoint":""} Apr 16 16:03:04.342950 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:04.342925 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Apr 16 16:03:04.342950 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:04.342955 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Apr 16 16:03:04.415410 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:04.414894 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:04.415410 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:04.414918 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:04.415410 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:04.415036 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:03:04.415410 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:04.415160 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:03:04.509682 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:04.509560 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m98lv" event={"ID":"d44ec06b-81b1-4bac-9abd-8efadd98b94a","Type":"ContainerStarted","Data":"ababb104d5e2713f76ca1eda79a8779f6ca6257a25cefde6020e0615609313cf"} Apr 16 16:03:04.511782 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:04.511754 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Apr 16 16:03:04.511912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:04.511797 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" event={"ID":"0369ef86-452e-4148-ab71-6ccca93defc5","Type":"ContainerStarted","Data":"11269a5db03409f8e9d759ed22be4bb5ee7474c89f4ddec6b5841505ead2bdea"} Apr 16 16:03:04.537754 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:04.537711 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m98lv" podStartSLOduration=5.525116199 podStartE2EDuration="22.537698538s" podCreationTimestamp="2026-04-16 16:02:42 +0000 UTC" firstStartedPulling="2026-04-16 16:02:45.129960013 +0000 UTC m=+3.388706815" lastFinishedPulling="2026-04-16 16:03:02.142542349 +0000 UTC m=+20.401289154" observedRunningTime="2026-04-16 16:03:04.537396482 +0000 UTC m=+22.796143292" watchObservedRunningTime="2026-04-16 16:03:04.537698538 +0000 UTC m=+22.796445348" Apr 16 16:03:05.259290 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:05.259260 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:05.259470 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:05.259365 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:03:05.259470 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:05.259417 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret podName:9b57199f-943e-4015-b87b-a582c2c06494 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:13.259404253 +0000 UTC m=+31.518151042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret") pod "global-pull-secret-syncer-cpt7l" (UID: "9b57199f-943e-4015-b87b-a582c2c06494") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:03:05.414426 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:05.414393 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:05.414589 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:05.414507 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpt7l" podUID="9b57199f-943e-4015-b87b-a582c2c06494" Apr 16 16:03:05.516366 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:05.516339 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:03:05.516867 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:05.516798 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" event={"ID":"c78e2321-0a40-4d26-89b3-afd2d6eee0a6","Type":"ContainerStarted","Data":"6bd6f6ce9a6754f3fd694b335d28609bff38eb2113401818661c513784307af7"} Apr 16 16:03:05.519000 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:05.518931 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" event={"ID":"0369ef86-452e-4148-ab71-6ccca93defc5","Type":"ContainerStarted","Data":"5b91172a8c480c0ca1f6bc8f3db80dc3fa772b53f644f9baa83fc1eb5ffa523c"} Apr 16 16:03:06.414345 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:06.414310 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:06.414533 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:06.414443 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:03:06.414533 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:06.414514 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:06.414674 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:06.414636 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:03:07.414601 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:07.414583 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:07.415095 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:07.414691 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpt7l" podUID="9b57199f-943e-4015-b87b-a582c2c06494" Apr 16 16:03:07.528789 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:07.528769 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:03:07.529127 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:07.529106 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" event={"ID":"c78e2321-0a40-4d26-89b3-afd2d6eee0a6","Type":"ContainerStarted","Data":"386bda62612c14c9735c81f195b6e2ebb0577b5fdcdd40012a91eff9f6bc70b4"} Apr 16 16:03:07.529562 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:07.529532 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:03:07.529562 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:07.529561 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:03:07.529713 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:07.529571 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:03:07.529764 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:07.529724 2576 scope.go:117] "RemoveContainer" containerID="ce18aa0e88969c3fd61d2a327769cb9e83ce1431744ac2419a98ba4f80783dd9" Apr 16 16:03:07.545059 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:07.545037 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:03:07.546146 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:07.545500 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:03:07.565411 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:07.565363 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8z2lq" podStartSLOduration=5.557160387 podStartE2EDuration="25.565346735s" podCreationTimestamp="2026-04-16 16:02:42 +0000 UTC" firstStartedPulling="2026-04-16 16:02:45.127215211 +0000 UTC m=+3.385962000" lastFinishedPulling="2026-04-16 16:03:05.135401549 +0000 UTC m=+23.394148348" observedRunningTime="2026-04-16 16:03:05.539335358 +0000 UTC m=+23.798082170" watchObservedRunningTime="2026-04-16 16:03:07.565346735 +0000 UTC m=+25.824093586" Apr 16 16:03:08.414973 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:08.414943 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:08.415353 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:08.414943 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:08.415353 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:08.415079 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:03:08.415353 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:08.415132 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:03:08.532116 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:08.532090 2576 generic.go:358] "Generic (PLEG): container finished" podID="4bb6679c-eb46-4326-bf8e-30829f808afc" containerID="f6d484eb111d3a7bcb420eeb0f78fdf436de85e7ee8abf0b4a7ac805e9255dec" exitCode=0 Apr 16 16:03:08.532232 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:08.532163 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" event={"ID":"4bb6679c-eb46-4326-bf8e-30829f808afc","Type":"ContainerDied","Data":"f6d484eb111d3a7bcb420eeb0f78fdf436de85e7ee8abf0b4a7ac805e9255dec"} Apr 16 16:03:08.535395 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:08.535376 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:03:08.535705 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:08.535686 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" event={"ID":"c78e2321-0a40-4d26-89b3-afd2d6eee0a6","Type":"ContainerStarted","Data":"f8bf65edefe534229e40a98bb287fc85aba1a33d509e2bc591b9ffb9befbd346"} Apr 16 16:03:08.588900 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:08.588846 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" podStartSLOduration=9.49660957 podStartE2EDuration="26.588831405s" podCreationTimestamp="2026-04-16 16:02:42 +0000 UTC" firstStartedPulling="2026-04-16 16:02:45.132147625 +0000 UTC m=+3.390894428" lastFinishedPulling="2026-04-16 16:03:02.224369457 +0000 UTC m=+20.483116263" observedRunningTime="2026-04-16 16:03:08.588429026 +0000 UTC m=+26.847175836" watchObservedRunningTime="2026-04-16 16:03:08.588831405 +0000 UTC m=+26.847578214" Apr 16 16:03:09.334805 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:09.334623 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2lh6g"] Apr 16 16:03:09.334970 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:09.334889 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:09.335081 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:09.335061 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:03:09.337106 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:09.337081 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fpztb"] Apr 16 16:03:09.337189 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:09.337161 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:09.337250 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:09.337234 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:03:09.346814 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:09.346792 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cpt7l"] Apr 16 16:03:09.346915 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:09.346880 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:09.346953 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:09.346942 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpt7l" podUID="9b57199f-943e-4015-b87b-a582c2c06494" Apr 16 16:03:09.539144 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:09.539111 2576 generic.go:358] "Generic (PLEG): container finished" podID="4bb6679c-eb46-4326-bf8e-30829f808afc" containerID="84bdc51f322175143e2b698993982a7b49f981e81f5543017e8ca7d67578d7ac" exitCode=0 Apr 16 16:03:09.539521 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:09.539151 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" event={"ID":"4bb6679c-eb46-4326-bf8e-30829f808afc","Type":"ContainerDied","Data":"84bdc51f322175143e2b698993982a7b49f981e81f5543017e8ca7d67578d7ac"} Apr 16 16:03:10.414495 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:10.414462 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:10.414609 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:10.414563 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:03:10.543559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:10.543526 2576 generic.go:358] "Generic (PLEG): container finished" podID="4bb6679c-eb46-4326-bf8e-30829f808afc" containerID="2b39c4bfa1bfbfe1fe9434e539b3ef5cff57a41dc4f3678c04f73fa97115ce3b" exitCode=0 Apr 16 16:03:10.543559 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:10.543564 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" event={"ID":"4bb6679c-eb46-4326-bf8e-30829f808afc","Type":"ContainerDied","Data":"2b39c4bfa1bfbfe1fe9434e539b3ef5cff57a41dc4f3678c04f73fa97115ce3b"} Apr 16 16:03:11.414672 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:11.414643 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:11.414836 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:11.414643 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:11.414836 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:11.414756 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpt7l" podUID="9b57199f-943e-4015-b87b-a582c2c06494" Apr 16 16:03:11.414945 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:11.414868 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:03:12.254450 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:12.254414 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lg294" Apr 16 16:03:12.255165 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:12.254566 2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Apr 16 16:03:12.255340 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:12.255323 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lg294" Apr 16 16:03:12.415347 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:12.415324 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:12.415522 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:12.415407 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:03:13.320515 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:13.320429 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:13.321127 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:13.320550 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Apr 16 16:03:13.321127 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:13.320595 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret podName:9b57199f-943e-4015-b87b-a582c2c06494 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:29.320582758 +0000 UTC m=+47.579329546 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret") pod "global-pull-secret-syncer-cpt7l" (UID: "9b57199f-943e-4015-b87b-a582c2c06494") : object "kube-system"/"original-pull-secret" not registered Apr 16 16:03:13.414906 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:13.414870 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:13.414906 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:13.414904 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:13.415160 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:13.414995 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:03:13.415160 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:13.415106 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpt7l" podUID="9b57199f-943e-4015-b87b-a582c2c06494" Apr 16 16:03:14.414991 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:14.414960 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:14.415491 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:14.415096 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2lh6g" podUID="23dd67cf-2731-4685-80f7-f4a1e75b6476" Apr 16 16:03:15.414967 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.414882 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:15.415151 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.414882 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:15.415151 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:15.415038 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cpt7l" podUID="9b57199f-943e-4015-b87b-a582c2c06494" Apr 16 16:03:15.415681 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:15.415152 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:03:15.550819 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.550786 2576 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-143-177.ec2.internal" event="NodeReady" Apr 16 16:03:15.550990 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.550923 2576 kubelet_node_status.go:550] "Fast updating node status as it just became ready" Apr 16 16:03:15.591821 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.591788 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-58d7958c46-l2zxs"] Apr 16 16:03:15.597028 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.596986 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.600006 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.599982 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5j7ls\"" Apr 16 16:03:15.600135 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.600060 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Apr 16 16:03:15.600361 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.600345 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Apr 16 16:03:15.600431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.600355 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Apr 16 16:03:15.608405 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.608383 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Apr 16 16:03:15.612867 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.612848 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mt4rq"] Apr 16 16:03:15.616582 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.616562 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-58d7958c46-l2zxs"] Apr 16 16:03:15.616722 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.616675 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.619160 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.619037 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Apr 16 16:03:15.619160 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.619049 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jhn24\"" Apr 16 16:03:15.619717 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.619697 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Apr 16 16:03:15.621234 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.621213 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kmvkl"] Apr 16 16:03:15.623904 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.623885 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:15.626334 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.626314 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mt4rq"] Apr 16 16:03:15.628647 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.628630 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Apr 16 16:03:15.628781 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.628759 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l2qxk\"" Apr 16 16:03:15.629002 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.628986 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Apr 16 16:03:15.633390 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.633290 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Apr 16 16:03:15.655405 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.655381 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kmvkl"] Apr 16 16:03:15.740262 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740225 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-installation-pull-secrets\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.740262 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740267 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7697a0-8cda-49df-861c-5762c30a7794-tmp-dir\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.740489 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-trusted-ca\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.740489 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc5c8\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-kube-api-access-qc5c8\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.740489 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740438 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:15.740618 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5zmz\" (UniqueName: \"kubernetes.io/projected/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-kube-api-access-m5zmz\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:15.740618 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7697a0-8cda-49df-861c-5762c30a7794-config-volume\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.740618 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740588 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.740734 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4fq\" (UniqueName: \"kubernetes.io/projected/ad7697a0-8cda-49df-861c-5762c30a7794-kube-api-access-4d4fq\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.740734 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740670 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ed80516-2524-487c-8c72-d7ded9a82159-ca-trust-extracted\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.740734 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740694 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-registry-certificates\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.740734 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-bound-sa-token\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.740877 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-image-registry-private-configuration\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.740877 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.740803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.842132 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842093 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.842319 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842145 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-installation-pull-secrets\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.842319 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842168 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7697a0-8cda-49df-861c-5762c30a7794-tmp-dir\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.842319 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842196 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-trusted-ca\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.842319 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842234 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qc5c8\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-kube-api-access-qc5c8\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.842319 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842255 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:15.842319 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:15.842271 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:03:15.842319 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842284 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m5zmz\" (UniqueName: \"kubernetes.io/projected/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-kube-api-access-m5zmz\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:15.842319 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842309 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7697a0-8cda-49df-861c-5762c30a7794-config-volume\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.842711 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:15.842343 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls podName:ad7697a0-8cda-49df-861c-5762c30a7794 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:16.342324189 +0000 UTC m=+34.601071004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls") pod "dns-default-mt4rq" (UID: "ad7697a0-8cda-49df-861c-5762c30a7794") : secret "dns-default-metrics-tls" not found Apr 16 16:03:15.842711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842367 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.842711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842424 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4fq\" (UniqueName: \"kubernetes.io/projected/ad7697a0-8cda-49df-861c-5762c30a7794-kube-api-access-4d4fq\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.842711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842467 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ed80516-2524-487c-8c72-d7ded9a82159-ca-trust-extracted\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.842711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842493 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-registry-certificates\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.842711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842520 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-bound-sa-token\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.842711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842553 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-image-registry-private-configuration\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.842711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842561 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad7697a0-8cda-49df-861c-5762c30a7794-tmp-dir\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.843116 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842912 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ed80516-2524-487c-8c72-d7ded9a82159-ca-trust-extracted\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.843116 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.842951 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ad7697a0-8cda-49df-861c-5762c30a7794-config-volume\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.843116 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:15.842958 2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 16:03:15.843116 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:15.842983 2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7958c46-l2zxs: secret "image-registry-tls" not found Apr 16 16:03:15.843304 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:15.843237 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:03:15.843304 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:15.843295 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert podName:f4bb9693-c1c3-4ecc-b82d-2ea9913caa85 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:16.343278845 +0000 UTC m=+34.602025653 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert") pod "ingress-canary-kmvkl" (UID: "f4bb9693-c1c3-4ecc-b82d-2ea9913caa85") : secret "canary-serving-cert" not found Apr 16 16:03:15.843430 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.843407 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-registry-certificates\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.843490 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:15.843449 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls podName:7ed80516-2524-487c-8c72-d7ded9a82159 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:16.343433741 +0000 UTC m=+34.602180553 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls") pod "image-registry-58d7958c46-l2zxs" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159") : secret "image-registry-tls" not found Apr 16 16:03:15.844187 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.844137 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-trusted-ca\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.847863 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.847817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-image-registry-private-configuration\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.847863 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.847820 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-installation-pull-secrets\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.853425 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.852966 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4fq\" (UniqueName: \"kubernetes.io/projected/ad7697a0-8cda-49df-861c-5762c30a7794-kube-api-access-4d4fq\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:15.853728 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.853682 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5zmz\" (UniqueName: \"kubernetes.io/projected/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-kube-api-access-m5zmz\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:15.853728 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.853699 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc5c8\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-kube-api-access-qc5c8\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:15.853728 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:15.853722 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-bound-sa-token\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:16.145654 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:16.145630 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:16.145752 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:16.145692 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bfj\" (UniqueName: \"kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj\") pod \"network-check-target-2lh6g\" (UID: \"23dd67cf-2731-4685-80f7-f4a1e75b6476\") " pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:16.145797 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.145778 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:03:16.145834 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.145827 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs podName:cd5f8939-2604-4ca4-8d1e-05dc7f86fa41 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:48.145812536 +0000 UTC m=+66.404559334 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs") pod "network-metrics-daemon-fpztb" (UID: "cd5f8939-2604-4ca4-8d1e-05dc7f86fa41") : object "openshift-multus"/"metrics-daemon-secret" not registered Apr 16 16:03:16.145880 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.145824 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Apr 16 16:03:16.145880 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.145851 2576 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Apr 16 16:03:16.145880 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.145867 2576 projected.go:194] Error preparing data for projected volume kube-api-access-q8bfj for pod openshift-network-diagnostics/network-check-target-2lh6g: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:03:16.145989 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.145926 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj podName:23dd67cf-2731-4685-80f7-f4a1e75b6476 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:48.145908832 +0000 UTC m=+66.404655643 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-q8bfj" (UniqueName: "kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj") pod "network-check-target-2lh6g" (UID: "23dd67cf-2731-4685-80f7-f4a1e75b6476") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Apr 16 16:03:16.347065 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:16.347033 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:16.347178 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:16.347085 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:16.347178 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:16.347111 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:16.347255 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.347174 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:03:16.347255 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.347192 2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 16:03:16.347255 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.347202 2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7958c46-l2zxs: secret "image-registry-tls" not found Apr 16 16:03:16.347255 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.347220 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:03:16.347255 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.347243 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls podName:ad7697a0-8cda-49df-861c-5762c30a7794 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:17.347225985 +0000 UTC m=+35.605972807 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls") pod "dns-default-mt4rq" (UID: "ad7697a0-8cda-49df-861c-5762c30a7794") : secret "dns-default-metrics-tls" not found Apr 16 16:03:16.347404 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.347264 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert podName:f4bb9693-c1c3-4ecc-b82d-2ea9913caa85 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:17.347250231 +0000 UTC m=+35.605997059 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert") pod "ingress-canary-kmvkl" (UID: "f4bb9693-c1c3-4ecc-b82d-2ea9913caa85") : secret "canary-serving-cert" not found Apr 16 16:03:16.347404 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:16.347277 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls podName:7ed80516-2524-487c-8c72-d7ded9a82159 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:17.347270973 +0000 UTC m=+35.606017761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls") pod "image-registry-58d7958c46-l2zxs" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159") : secret "image-registry-tls" not found Apr 16 16:03:16.413985 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:16.413955 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:16.419416 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:16.419398 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qf27n\"" Apr 16 16:03:16.419774 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:16.419398 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 16 16:03:16.419774 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:16.419464 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 16 16:03:16.556880 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:16.556808 2576 generic.go:358] "Generic (PLEG): container finished" podID="4bb6679c-eb46-4326-bf8e-30829f808afc" containerID="32e64ab611edeb5ce03c03878fec151ebfa0ed101f87a615b1af4b1cce8c4f92" exitCode=0 Apr 16 16:03:16.556880 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:16.556865 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" event={"ID":"4bb6679c-eb46-4326-bf8e-30829f808afc","Type":"ContainerDied","Data":"32e64ab611edeb5ce03c03878fec151ebfa0ed101f87a615b1af4b1cce8c4f92"} Apr 16 16:03:17.358367 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:17.358338 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:17.358550 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:17.358400 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:17.358550 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:17.358469 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:17.358550 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:17.358483 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:03:17.358550 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:17.358541 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert podName:f4bb9693-c1c3-4ecc-b82d-2ea9913caa85 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:19.358527061 +0000 UTC m=+37.617273849 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert") pod "ingress-canary-kmvkl" (UID: "f4bb9693-c1c3-4ecc-b82d-2ea9913caa85") : secret "canary-serving-cert" not found Apr 16 16:03:17.358722 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:17.358564 2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 16:03:17.358722 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:17.358574 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:03:17.358722 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:17.358584 2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7958c46-l2zxs: secret "image-registry-tls" not found Apr 16 16:03:17.358722 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:17.358632 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls podName:ad7697a0-8cda-49df-861c-5762c30a7794 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:19.358616825 +0000 UTC m=+37.617363619 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls") pod "dns-default-mt4rq" (UID: "ad7697a0-8cda-49df-861c-5762c30a7794") : secret "dns-default-metrics-tls" not found Apr 16 16:03:17.358722 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:17.358650 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls podName:7ed80516-2524-487c-8c72-d7ded9a82159 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:19.358639226 +0000 UTC m=+37.617386021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls") pod "image-registry-58d7958c46-l2zxs" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159") : secret "image-registry-tls" not found Apr 16 16:03:17.414574 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:17.414549 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:17.414688 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:17.414554 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:17.417859 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:17.417840 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hwq6n\"" Apr 16 16:03:17.418901 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:17.418887 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 16 16:03:17.418970 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:17.418891 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Apr 16 16:03:17.561343 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:17.561314 2576 generic.go:358] "Generic (PLEG): container finished" podID="4bb6679c-eb46-4326-bf8e-30829f808afc" containerID="b171a3dc55197e9a072fb9926c5d91205c02ec1ec2bd57753bc7e9bf89962d59" exitCode=0 Apr 16 16:03:17.561710 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:17.561349 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" event={"ID":"4bb6679c-eb46-4326-bf8e-30829f808afc","Type":"ContainerDied","Data":"b171a3dc55197e9a072fb9926c5d91205c02ec1ec2bd57753bc7e9bf89962d59"} Apr 16 16:03:18.565924 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:18.565887 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" event={"ID":"4bb6679c-eb46-4326-bf8e-30829f808afc","Type":"ContainerStarted","Data":"d332a51864da7951c35bde4ed79f7083109a1a67fef91e2879ed8bfa3a1c5d0e"} Apr 16 16:03:18.605849 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:18.605798 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dfnlw" podStartSLOduration=5.771253275 podStartE2EDuration="36.605784328s" podCreationTimestamp="2026-04-16 16:02:42 +0000 UTC" firstStartedPulling="2026-04-16 16:02:45.134996509 +0000 UTC m=+3.393743307" lastFinishedPulling="2026-04-16 16:03:15.969527555 +0000 UTC m=+34.228274360" observedRunningTime="2026-04-16 16:03:18.60552243 +0000 UTC m=+36.864269239" watchObservedRunningTime="2026-04-16 16:03:18.605784328 +0000 UTC m=+36.864531191" Apr 16 16:03:19.375817 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:19.375786 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:19.375983 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:19.375843 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:19.375983 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:19.375885 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:19.375983 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:19.375934 2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 16:03:19.375983 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:19.375957 2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7958c46-l2zxs: secret "image-registry-tls" not found Apr 16 16:03:19.375983 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:19.375964 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:03:19.376177 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:19.375986 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:03:19.376177 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:19.376038 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls podName:7ed80516-2524-487c-8c72-d7ded9a82159 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:23.375997831 +0000 UTC m=+41.634744643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls") pod "image-registry-58d7958c46-l2zxs" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159") : secret "image-registry-tls" not found Apr 16 16:03:19.376177 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:19.376061 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls podName:ad7697a0-8cda-49df-861c-5762c30a7794 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:23.376047697 +0000 UTC m=+41.634794502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls") pod "dns-default-mt4rq" (UID: "ad7697a0-8cda-49df-861c-5762c30a7794") : secret "dns-default-metrics-tls" not found Apr 16 16:03:19.376177 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:19.376078 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert podName:f4bb9693-c1c3-4ecc-b82d-2ea9913caa85 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:23.376071456 +0000 UTC m=+41.634818244 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert") pod "ingress-canary-kmvkl" (UID: "f4bb9693-c1c3-4ecc-b82d-2ea9913caa85") : secret "canary-serving-cert" not found Apr 16 16:03:23.403964 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:23.403932 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:23.404377 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:23.403973 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:23.404377 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:23.404032 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:23.404377 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:23.404096 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:03:23.404377 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:23.404127 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:03:23.404377 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:23.404132 2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 16:03:23.404377 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:23.404146 2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7958c46-l2zxs: secret "image-registry-tls" not found Apr 16 16:03:23.404377 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:23.404160 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert podName:f4bb9693-c1c3-4ecc-b82d-2ea9913caa85 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:31.404142927 +0000 UTC m=+49.662889735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert") pod "ingress-canary-kmvkl" (UID: "f4bb9693-c1c3-4ecc-b82d-2ea9913caa85") : secret "canary-serving-cert" not found Apr 16 16:03:23.404377 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:23.404183 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls podName:7ed80516-2524-487c-8c72-d7ded9a82159 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:31.404168605 +0000 UTC m=+49.662915408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls") pod "image-registry-58d7958c46-l2zxs" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159") : secret "image-registry-tls" not found Apr 16 16:03:23.404377 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:23.404199 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls podName:ad7697a0-8cda-49df-861c-5762c30a7794 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:31.404191256 +0000 UTC m=+49.662938044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls") pod "dns-default-mt4rq" (UID: "ad7697a0-8cda-49df-861c-5762c30a7794") : secret "dns-default-metrics-tls" not found Apr 16 16:03:29.346170 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:29.346131 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:29.348924 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:29.348896 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9b57199f-943e-4015-b87b-a582c2c06494-original-pull-secret\") pod \"global-pull-secret-syncer-cpt7l\" (UID: \"9b57199f-943e-4015-b87b-a582c2c06494\") " pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:29.434924 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:29.434900 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cpt7l" Apr 16 16:03:29.572466 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:29.572433 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cpt7l"] Apr 16 16:03:29.575723 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:03:29.575699 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b57199f_943e_4015_b87b_a582c2c06494.slice/crio-e5289d32f04daa018e392b08927799df5a67dd6d87c24565cf303e93073607a2 WatchSource:0}: Error finding container e5289d32f04daa018e392b08927799df5a67dd6d87c24565cf303e93073607a2: Status 404 returned error can't find the container with id e5289d32f04daa018e392b08927799df5a67dd6d87c24565cf303e93073607a2 Apr 16 16:03:29.587836 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:29.587812 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cpt7l" event={"ID":"9b57199f-943e-4015-b87b-a582c2c06494","Type":"ContainerStarted","Data":"e5289d32f04daa018e392b08927799df5a67dd6d87c24565cf303e93073607a2"} Apr 16 16:03:31.460068 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:31.460010 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:31.460493 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:31.460108 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:31.460493 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:31.460157 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:31.460493 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:31.460187 2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 16:03:31.460493 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:31.460212 2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7958c46-l2zxs: secret "image-registry-tls" not found Apr 16 16:03:31.460493 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:31.460249 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:03:31.460493 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:31.460249 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:03:31.460493 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:31.460330 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls podName:7ed80516-2524-487c-8c72-d7ded9a82159 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:47.460260031 +0000 UTC m=+65.719006833 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls") pod "image-registry-58d7958c46-l2zxs" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159") : secret "image-registry-tls" not found Apr 16 16:03:31.460493 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:31.460354 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert podName:f4bb9693-c1c3-4ecc-b82d-2ea9913caa85 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:47.460342582 +0000 UTC m=+65.719089376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert") pod "ingress-canary-kmvkl" (UID: "f4bb9693-c1c3-4ecc-b82d-2ea9913caa85") : secret "canary-serving-cert" not found Apr 16 16:03:31.460493 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:31.460375 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls podName:ad7697a0-8cda-49df-861c-5762c30a7794 nodeName:}" failed. No retries permitted until 2026-04-16 16:03:47.460366572 +0000 UTC m=+65.719113363 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls") pod "dns-default-mt4rq" (UID: "ad7697a0-8cda-49df-861c-5762c30a7794") : secret "dns-default-metrics-tls" not found Apr 16 16:03:34.598310 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:34.598270 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cpt7l" event={"ID":"9b57199f-943e-4015-b87b-a582c2c06494","Type":"ContainerStarted","Data":"5089c5f1b4be0f0e740d9c074560e4eebc0b24b71eccbdda504352811fd298b2"} Apr 16 16:03:39.550000 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:39.549961 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gtjzw" Apr 16 16:03:39.585712 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:39.585667 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cpt7l" podStartSLOduration=38.584737703 podStartE2EDuration="42.585655448s" podCreationTimestamp="2026-04-16 16:02:57 +0000 UTC" firstStartedPulling="2026-04-16 16:03:29.577202584 +0000 UTC m=+47.835949376" lastFinishedPulling="2026-04-16 16:03:33.57812033 +0000 UTC m=+51.836867121" observedRunningTime="2026-04-16 16:03:34.619268819 +0000 UTC m=+52.878015629" watchObservedRunningTime="2026-04-16 16:03:39.585655448 +0000 UTC m=+57.844402257" Apr 16 16:03:47.479466 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:47.479431 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:03:47.479870 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:47.479481 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:03:47.479870 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:47.479786 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:03:47.479982 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:47.479915 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:03:47.479982 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:47.479967 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:03:47.480196 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:47.480178 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls podName:ad7697a0-8cda-49df-861c-5762c30a7794 nodeName:}" failed. No retries permitted until 2026-04-16 16:04:19.480153176 +0000 UTC m=+97.738899982 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls") pod "dns-default-mt4rq" (UID: "ad7697a0-8cda-49df-861c-5762c30a7794") : secret "dns-default-metrics-tls" not found Apr 16 16:03:47.480260 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:47.480213 2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 16:03:47.480260 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:47.480227 2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7958c46-l2zxs: secret "image-registry-tls" not found Apr 16 16:03:47.480472 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:47.480459 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls podName:7ed80516-2524-487c-8c72-d7ded9a82159 nodeName:}" failed. No retries permitted until 2026-04-16 16:04:19.480441876 +0000 UTC m=+97.739188683 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls") pod "image-registry-58d7958c46-l2zxs" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159") : secret "image-registry-tls" not found Apr 16 16:03:47.480630 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:47.480615 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert podName:f4bb9693-c1c3-4ecc-b82d-2ea9913caa85 nodeName:}" failed. No retries permitted until 2026-04-16 16:04:19.480591107 +0000 UTC m=+97.739337912 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert") pod "ingress-canary-kmvkl" (UID: "f4bb9693-c1c3-4ecc-b82d-2ea9913caa85") : secret "canary-serving-cert" not found Apr 16 16:03:48.185607 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:48.185574 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:03:48.185766 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:48.185631 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q8bfj\" (UniqueName: \"kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj\") pod \"network-check-target-2lh6g\" (UID: \"23dd67cf-2731-4685-80f7-f4a1e75b6476\") " pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:48.188383 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:48.188362 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Apr 16 16:03:48.188451 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:48.188435 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Apr 16 16:03:48.195811 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:48.195793 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 16 16:03:48.195885 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:03:48.195873 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs podName:cd5f8939-2604-4ca4-8d1e-05dc7f86fa41 nodeName:}" failed. No retries permitted until 2026-04-16 16:04:52.195853579 +0000 UTC m=+130.454600389 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs") pod "network-metrics-daemon-fpztb" (UID: "cd5f8939-2604-4ca4-8d1e-05dc7f86fa41") : secret "metrics-daemon-secret" not found Apr 16 16:03:48.198600 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:48.198584 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Apr 16 16:03:48.208495 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:48.208474 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8bfj\" (UniqueName: \"kubernetes.io/projected/23dd67cf-2731-4685-80f7-f4a1e75b6476-kube-api-access-q8bfj\") pod \"network-check-target-2lh6g\" (UID: \"23dd67cf-2731-4685-80f7-f4a1e75b6476\") " pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:48.239096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:48.239071 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qf27n\"" Apr 16 16:03:48.247299 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:48.247278 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:48.377449 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:48.377398 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2lh6g"] Apr 16 16:03:48.380479 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:03:48.380457 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23dd67cf_2731_4685_80f7_f4a1e75b6476.slice/crio-e4a72b18d777dc897b6ad93bebac06141c4ab64aec172b78593708e79bcc0c27 WatchSource:0}: Error finding container e4a72b18d777dc897b6ad93bebac06141c4ab64aec172b78593708e79bcc0c27: Status 404 returned error can't find the container with id e4a72b18d777dc897b6ad93bebac06141c4ab64aec172b78593708e79bcc0c27 Apr 16 16:03:48.624594 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:48.624563 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2lh6g" event={"ID":"23dd67cf-2731-4685-80f7-f4a1e75b6476","Type":"ContainerStarted","Data":"e4a72b18d777dc897b6ad93bebac06141c4ab64aec172b78593708e79bcc0c27"} Apr 16 16:03:51.631307 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:51.631274 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2lh6g" event={"ID":"23dd67cf-2731-4685-80f7-f4a1e75b6476","Type":"ContainerStarted","Data":"63920936a1a2d0079ea15f7c01f83888af5a5f57ac10977181a964850d2c2351"} Apr 16 16:03:51.631787 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:51.631407 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:03:51.648128 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:03:51.648083 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2lh6g" podStartSLOduration=67.033914878 podStartE2EDuration="1m9.648070419s" podCreationTimestamp="2026-04-16 16:02:42 +0000 UTC" firstStartedPulling="2026-04-16 16:03:48.382807515 +0000 UTC m=+66.641554306" lastFinishedPulling="2026-04-16 16:03:50.996963056 +0000 UTC m=+69.255709847" observedRunningTime="2026-04-16 16:03:51.647841568 +0000 UTC m=+69.906588377" watchObservedRunningTime="2026-04-16 16:03:51.648070419 +0000 UTC m=+69.906817230" Apr 16 16:04:19.519522 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:04:19.519475 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:04:19.519522 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:04:19.519535 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:04:19.519933 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:04:19.519573 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:04:19.519933 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:04:19.519622 2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 16:04:19.519933 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:04:19.519643 2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7958c46-l2zxs: secret "image-registry-tls" not found Apr 16 16:04:19.519933 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:04:19.519659 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:04:19.519933 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:04:19.519668 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:04:19.519933 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:04:19.519708 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls podName:7ed80516-2524-487c-8c72-d7ded9a82159 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:23.519692502 +0000 UTC m=+161.778439294 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls") pod "image-registry-58d7958c46-l2zxs" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159") : secret "image-registry-tls" not found Apr 16 16:04:19.519933 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:04:19.519722 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls podName:ad7697a0-8cda-49df-861c-5762c30a7794 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:23.519715919 +0000 UTC m=+161.778462707 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls") pod "dns-default-mt4rq" (UID: "ad7697a0-8cda-49df-861c-5762c30a7794") : secret "dns-default-metrics-tls" not found Apr 16 16:04:19.519933 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:04:19.519731 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert podName:f4bb9693-c1c3-4ecc-b82d-2ea9913caa85 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:23.519726461 +0000 UTC m=+161.778473249 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert") pod "ingress-canary-kmvkl" (UID: "f4bb9693-c1c3-4ecc-b82d-2ea9913caa85") : secret "canary-serving-cert" not found Apr 16 16:04:22.635659 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:04:22.635622 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2lh6g" Apr 16 16:04:52.239187 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:04:52.239132 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:04:52.239674 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:04:52.239360 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Apr 16 16:04:52.239674 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:04:52.239448 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs podName:cd5f8939-2604-4ca4-8d1e-05dc7f86fa41 nodeName:}" failed. No retries permitted until 2026-04-16 16:06:54.239420396 +0000 UTC m=+252.498167185 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs") pod "network-metrics-daemon-fpztb" (UID: "cd5f8939-2604-4ca4-8d1e-05dc7f86fa41") : secret "metrics-daemon-secret" not found Apr 16 16:05:18.503383 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.503342 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-d87b8d5fc-qb2g6"] Apr 16 16:05:18.504939 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.504921 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.507524 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.507503 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Apr 16 16:05:18.507636 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.507607 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9q8ww\"" Apr 16 16:05:18.508892 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.508874 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Apr 16 16:05:18.508982 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.508874 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Apr 16 16:05:18.508982 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.508900 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Apr 16 16:05:18.515408 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.515390 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Apr 16 16:05:18.521172 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.521148 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-d87b8d5fc-qb2g6"] Apr 16 16:05:18.600169 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.600136 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7d955d5dd4-k947k"] Apr 16 16:05:18.601885 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.601871 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-7d955d5dd4-k947k" Apr 16 16:05:18.603820 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.603780 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr"] Apr 16 16:05:18.605322 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.605300 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Apr 16 16:05:18.605433 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.605357 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Apr 16 16:05:18.605491 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.605478 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2"] Apr 16 16:05:18.605628 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.605613 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" Apr 16 16:05:18.605863 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.605842 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7vrmg\"" Apr 16 16:05:18.606956 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.606940 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5785d4fcdd-q2snw"] Apr 16 16:05:18.607105 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.607092 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:18.608534 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.608520 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.611242 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.611225 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Apr 16 16:05:18.611502 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.611483 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Apr 16 16:05:18.611652 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.611630 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" podUID="7ed80516-2524-487c-8c72-d7ded9a82159" Apr 16 16:05:18.611759 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.611734 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Apr 16 16:05:18.611835 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.611769 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Apr 16 16:05:18.611835 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.611783 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-zpg7v\"" Apr 16 16:05:18.611835 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.611786 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pblvm\"" Apr 16 16:05:18.612137 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.612121 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Apr 16 16:05:18.613555 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.613531 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r52hg\" (UniqueName: \"kubernetes.io/projected/9541790d-c119-472d-ab6a-4915d4065240-kube-api-access-r52hg\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.613711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.613687 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9541790d-c119-472d-ab6a-4915d4065240-serving-cert\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.613828 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.613764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9541790d-c119-472d-ab6a-4915d4065240-config\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.613885 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.613871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9541790d-c119-472d-ab6a-4915d4065240-trusted-ca\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.614793 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.614611 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Apr 16 16:05:18.614793 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.614621 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Apr 16 16:05:18.614793 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.614659 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Apr 16 16:05:18.614793 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.614693 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7d955d5dd4-k947k"] Apr 16 16:05:18.616393 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.616209 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Apr 16 16:05:18.616702 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.616681 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Apr 16 16:05:18.617208 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.617193 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Apr 16 16:05:18.617621 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.617489 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Apr 16 16:05:18.617848 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.617829 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2"] Apr 16 16:05:18.618403 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.618381 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-r62t8\"" Apr 16 16:05:18.621727 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.621706 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr"] Apr 16 16:05:18.622811 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.622791 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5785d4fcdd-q2snw"] Apr 16 16:05:18.625910 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.625888 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Apr 16 16:05:18.627592 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.627566 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mt4rq" podUID="ad7697a0-8cda-49df-861c-5762c30a7794" Apr 16 16:05:18.636839 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.636813 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-kmvkl" podUID="f4bb9693-c1c3-4ecc-b82d-2ea9913caa85" Apr 16 16:05:18.701425 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.701395 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c"] Apr 16 16:05:18.703218 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.703200 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:18.706321 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.706301 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Apr 16 16:05:18.706321 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.706319 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Apr 16 16:05:18.706535 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.706525 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-74jmg\"" Apr 16 16:05:18.707324 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.707307 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj"] Apr 16 16:05:18.713271 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.713252 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6fdcb8c7b8-jprpz"] Apr 16 16:05:18.713421 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.713406 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:18.714864 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.714843 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7d332411-d30c-4247-a333-0a0f77355443-telemetry-config\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:18.714976 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.714872 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qd9n\" (UniqueName: \"kubernetes.io/projected/057a2387-92a7-47ee-8fbd-ff2d0c13ee1a-kube-api-access-9qd9n\") pod \"kube-storage-version-migrator-operator-756bb7d76f-htwnr\" (UID: \"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" Apr 16 16:05:18.714976 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.714892 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbf86cc5-0e80-47e9-b081-8510a5c57973-trusted-ca-bundle\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.714976 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.714913 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r52hg\" (UniqueName: \"kubernetes.io/projected/9541790d-c119-472d-ab6a-4915d4065240-kube-api-access-r52hg\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.714976 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.714929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbf86cc5-0e80-47e9-b081-8510a5c57973-tmp\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.715210 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.714970 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9541790d-c119-472d-ab6a-4915d4065240-serving-cert\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.715210 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715008 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.715210 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715075 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbf86cc5-0e80-47e9-b081-8510a5c57973-serving-cert\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.715210 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715119 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9541790d-c119-472d-ab6a-4915d4065240-config\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.715210 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbf86cc5-0e80-47e9-b081-8510a5c57973-service-ca-bundle\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.715210 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715174 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dbf86cc5-0e80-47e9-b081-8510a5c57973-snapshots\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.715477 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715220 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:18.715477 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715254 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9541790d-c119-472d-ab6a-4915d4065240-trusted-ca\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.715477 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715290 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/057a2387-92a7-47ee-8fbd-ff2d0c13ee1a-config\") pod \"kube-storage-version-migrator-operator-756bb7d76f-htwnr\" (UID: \"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" Apr 16 16:05:18.715477 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9nzm\" (UniqueName: \"kubernetes.io/projected/0c553c40-80b0-4906-9c86-e744ca5c2eba-kube-api-access-g9nzm\") pod \"volume-data-source-validator-7d955d5dd4-k947k\" (UID: \"0c553c40-80b0-4906-9c86-e744ca5c2eba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7d955d5dd4-k947k" Apr 16 16:05:18.715477 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwwm7\" (UniqueName: \"kubernetes.io/projected/7d332411-d30c-4247-a333-0a0f77355443-kube-api-access-gwwm7\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:18.715727 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/057a2387-92a7-47ee-8fbd-ff2d0c13ee1a-serving-cert\") pod \"kube-storage-version-migrator-operator-756bb7d76f-htwnr\" (UID: \"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" Apr 16 16:05:18.715727 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xczp8\" (UniqueName: \"kubernetes.io/projected/dbf86cc5-0e80-47e9-b081-8510a5c57973-kube-api-access-xczp8\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.715840 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.715818 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9541790d-c119-472d-ab6a-4915d4065240-config\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.716148 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.716127 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9541790d-c119-472d-ab6a-4915d4065240-trusted-ca\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.717411 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.717392 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9541790d-c119-472d-ab6a-4915d4065240-serving-cert\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.722590 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.722571 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Apr 16 16:05:18.723025 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.722988 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Apr 16 16:05:18.723025 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.723004 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bspsf\"" Apr 16 16:05:18.723911 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.723890 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c"] Apr 16 16:05:18.726862 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.726847 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Apr 16 16:05:18.728164 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.728146 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Apr 16 16:05:18.728301 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.728251 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Apr 16 16:05:18.728381 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.728300 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Apr 16 16:05:18.728713 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.728687 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-d7b7v\"" Apr 16 16:05:18.733616 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.733600 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Apr 16 16:05:18.733950 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.733935 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Apr 16 16:05:18.734050 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.733961 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Apr 16 16:05:18.741832 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.741813 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r52hg\" (UniqueName: \"kubernetes.io/projected/9541790d-c119-472d-ab6a-4915d4065240-kube-api-access-r52hg\") pod \"console-operator-d87b8d5fc-qb2g6\" (UID: \"9541790d-c119-472d-ab6a-4915d4065240\") " pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.750115 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.750095 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj"] Apr 16 16:05:18.754307 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.754266 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6fdcb8c7b8-jprpz"] Apr 16 16:05:18.796970 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.796947 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:05:18.796970 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.796962 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mt4rq" Apr 16 16:05:18.797155 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.796956 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:05:18.816294 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816266 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xczp8\" (UniqueName: \"kubernetes.io/projected/dbf86cc5-0e80-47e9-b081-8510a5c57973-kube-api-access-xczp8\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.816420 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816305 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7d332411-d30c-4247-a333-0a0f77355443-telemetry-config\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:18.816420 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816335 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:18.816507 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816337 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9qd9n\" (UniqueName: \"kubernetes.io/projected/057a2387-92a7-47ee-8fbd-ff2d0c13ee1a-kube-api-access-9qd9n\") pod \"kube-storage-version-migrator-operator-756bb7d76f-htwnr\" (UID: \"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" Apr 16 16:05:18.816542 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816531 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbf86cc5-0e80-47e9-b081-8510a5c57973-trusted-ca-bundle\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.816587 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816562 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbf86cc5-0e80-47e9-b081-8510a5c57973-tmp\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.816625 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e3c6810-602d-4451-a18d-2b7f8b3e6245-nginx-conf\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:18.816672 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.816672 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w58qf\" (UniqueName: \"kubernetes.io/projected/25e38cb7-4445-4ec1-8afb-75356dc1c085-kube-api-access-w58qf\") pod \"cluster-samples-operator-667775844f-28sdj\" (UID: \"25e38cb7-4445-4ec1-8afb-75356dc1c085\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:18.816764 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816735 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbf86cc5-0e80-47e9-b081-8510a5c57973-serving-cert\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.816814 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816776 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbf86cc5-0e80-47e9-b081-8510a5c57973-service-ca-bundle\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.816814 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816805 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dbf86cc5-0e80-47e9-b081-8510a5c57973-snapshots\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.816901 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-default-certificate\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.816901 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816886 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhjsn\" (UniqueName: \"kubernetes.io/projected/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-kube-api-access-hhjsn\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-stats-auth\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816953 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dbf86cc5-0e80-47e9-b081-8510a5c57973-tmp\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.816976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls\") pod \"cluster-samples-operator-667775844f-28sdj\" (UID: \"25e38cb7-4445-4ec1-8afb-75356dc1c085\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.817007 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.817076 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/057a2387-92a7-47ee-8fbd-ff2d0c13ee1a-config\") pod \"kube-storage-version-migrator-operator-756bb7d76f-htwnr\" (UID: \"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.817118 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.817167 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/7d332411-d30c-4247-a333-0a0f77355443-telemetry-config\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.817178 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls podName:7d332411-d30c-4247-a333-0a0f77355443 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:19.317159273 +0000 UTC m=+157.575906078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-w6kz2" (UID: "7d332411-d30c-4247-a333-0a0f77355443") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.817116 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g9nzm\" (UniqueName: \"kubernetes.io/projected/0c553c40-80b0-4906-9c86-e744ca5c2eba-kube-api-access-g9nzm\") pod \"volume-data-source-validator-7d955d5dd4-k947k\" (UID: \"0c553c40-80b0-4906-9c86-e744ca5c2eba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7d955d5dd4-k947k" Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.817297 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.817333 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gwwm7\" (UniqueName: \"kubernetes.io/projected/7d332411-d30c-4247-a333-0a0f77355443-kube-api-access-gwwm7\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:18.817447 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.817354 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/057a2387-92a7-47ee-8fbd-ff2d0c13ee1a-serving-cert\") pod \"kube-storage-version-migrator-operator-756bb7d76f-htwnr\" (UID: \"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" Apr 16 16:05:18.818097 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.817455 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbf86cc5-0e80-47e9-b081-8510a5c57973-trusted-ca-bundle\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.818097 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.817663 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dbf86cc5-0e80-47e9-b081-8510a5c57973-service-ca-bundle\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.818385 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.818365 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/057a2387-92a7-47ee-8fbd-ff2d0c13ee1a-config\") pod \"kube-storage-version-migrator-operator-756bb7d76f-htwnr\" (UID: \"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" Apr 16 16:05:18.818531 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.818513 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/dbf86cc5-0e80-47e9-b081-8510a5c57973-snapshots\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.819366 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.819347 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dbf86cc5-0e80-47e9-b081-8510a5c57973-serving-cert\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.819471 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.819455 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/057a2387-92a7-47ee-8fbd-ff2d0c13ee1a-serving-cert\") pod \"kube-storage-version-migrator-operator-756bb7d76f-htwnr\" (UID: \"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" Apr 16 16:05:18.826159 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.826139 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9nzm\" (UniqueName: \"kubernetes.io/projected/0c553c40-80b0-4906-9c86-e744ca5c2eba-kube-api-access-g9nzm\") pod \"volume-data-source-validator-7d955d5dd4-k947k\" (UID: \"0c553c40-80b0-4906-9c86-e744ca5c2eba\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-7d955d5dd4-k947k" Apr 16 16:05:18.826651 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.826633 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwwm7\" (UniqueName: \"kubernetes.io/projected/7d332411-d30c-4247-a333-0a0f77355443-kube-api-access-gwwm7\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:18.826777 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.826759 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qd9n\" (UniqueName: \"kubernetes.io/projected/057a2387-92a7-47ee-8fbd-ff2d0c13ee1a-kube-api-access-9qd9n\") pod \"kube-storage-version-migrator-operator-756bb7d76f-htwnr\" (UID: \"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" Apr 16 16:05:18.826827 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.826774 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xczp8\" (UniqueName: \"kubernetes.io/projected/dbf86cc5-0e80-47e9-b081-8510a5c57973-kube-api-access-xczp8\") pod \"insights-operator-5785d4fcdd-q2snw\" (UID: \"dbf86cc5-0e80-47e9-b081-8510a5c57973\") " pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.913201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.913169 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-7d955d5dd4-k947k" Apr 16 16:05:18.918079 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.918053 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-default-certificate\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.918192 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.918135 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hhjsn\" (UniqueName: \"kubernetes.io/projected/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-kube-api-access-hhjsn\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.918192 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.918171 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-stats-auth\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.918305 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.918198 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls\") pod \"cluster-samples-operator-667775844f-28sdj\" (UID: \"25e38cb7-4445-4ec1-8afb-75356dc1c085\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:18.918305 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.918242 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.918411 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.918378 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:18.919049 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.918542 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 16 16:05:18.919049 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.918620 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls podName:25e38cb7-4445-4ec1-8afb-75356dc1c085 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:19.41859899 +0000 UTC m=+157.677345791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls") pod "cluster-samples-operator-667775844f-28sdj" (UID: "25e38cb7-4445-4ec1-8afb-75356dc1c085") : secret "samples-operator-tls" not found Apr 16 16:05:18.919049 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.918675 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e3c6810-602d-4451-a18d-2b7f8b3e6245-nginx-conf\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:18.919049 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.918704 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.919049 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.918731 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w58qf\" (UniqueName: \"kubernetes.io/projected/25e38cb7-4445-4ec1-8afb-75356dc1c085-kube-api-access-w58qf\") pod \"cluster-samples-operator-667775844f-28sdj\" (UID: \"25e38cb7-4445-4ec1-8afb-75356dc1c085\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:18.919049 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.918554 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 16 16:05:18.919049 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.918884 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert podName:5e3c6810-602d-4451-a18d-2b7f8b3e6245 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:19.418867822 +0000 UTC m=+157.677614625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert") pod "networking-console-plugin-5cb6cf4cb4-j6j9c" (UID: "5e3c6810-602d-4451-a18d-2b7f8b3e6245") : secret "networking-console-plugin-cert" not found Apr 16 16:05:18.919049 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.918789 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 16:05:18.919049 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.918931 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs podName:66494738-0f0c-42ff-b8e6-c0961cc5f3c1 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:19.418918699 +0000 UTC m=+157.677665488 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs") pod "router-default-6fdcb8c7b8-jprpz" (UID: "66494738-0f0c-42ff-b8e6-c0961cc5f3c1") : secret "router-metrics-certs-default" not found Apr 16 16:05:18.919049 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:18.918985 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle podName:66494738-0f0c-42ff-b8e6-c0961cc5f3c1 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:19.418968987 +0000 UTC m=+157.677715789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle") pod "router-default-6fdcb8c7b8-jprpz" (UID: "66494738-0f0c-42ff-b8e6-c0961cc5f3c1") : configmap references non-existent config key: service-ca.crt Apr 16 16:05:18.920305 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.920281 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5e3c6810-602d-4451-a18d-2b7f8b3e6245-nginx-conf\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:18.920819 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.920787 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-default-certificate\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.920915 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.920887 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-stats-auth\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.922722 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.922704 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" Apr 16 16:05:18.933427 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.933408 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w58qf\" (UniqueName: \"kubernetes.io/projected/25e38cb7-4445-4ec1-8afb-75356dc1c085-kube-api-access-w58qf\") pod \"cluster-samples-operator-667775844f-28sdj\" (UID: \"25e38cb7-4445-4ec1-8afb-75356dc1c085\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:18.933590 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.933573 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhjsn\" (UniqueName: \"kubernetes.io/projected/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-kube-api-access-hhjsn\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:18.935250 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.935236 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" Apr 16 16:05:18.940933 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:18.940893 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-d87b8d5fc-qb2g6"] Apr 16 16:05:18.944359 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:05:18.944331 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9541790d_c119_472d_ab6a_4915d4065240.slice/crio-1f0a7dcd8034fc1a3bb87a9c912e9427640b2019784941764d6fc0078ab1413f WatchSource:0}: Error finding container 1f0a7dcd8034fc1a3bb87a9c912e9427640b2019784941764d6fc0078ab1413f: Status 404 returned error can't find the container with id 1f0a7dcd8034fc1a3bb87a9c912e9427640b2019784941764d6fc0078ab1413f Apr 16 16:05:19.048735 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.048701 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-7d955d5dd4-k947k"] Apr 16 16:05:19.051746 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:05:19.051716 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c553c40_80b0_4906_9c86_e744ca5c2eba.slice/crio-6de8902033d0be4e9eac0c56b100e7f918d4f446277d55d820927d35355eb03a WatchSource:0}: Error finding container 6de8902033d0be4e9eac0c56b100e7f918d4f446277d55d820927d35355eb03a: Status 404 returned error can't find the container with id 6de8902033d0be4e9eac0c56b100e7f918d4f446277d55d820927d35355eb03a Apr 16 16:05:19.068913 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.068820 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr"] Apr 16 16:05:19.071082 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:05:19.071055 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057a2387_92a7_47ee_8fbd_ff2d0c13ee1a.slice/crio-61913e32e1407ad649441688d4af385e833b9dac865fd68a409efc17c682467b WatchSource:0}: Error finding container 61913e32e1407ad649441688d4af385e833b9dac865fd68a409efc17c682467b: Status 404 returned error can't find the container with id 61913e32e1407ad649441688d4af385e833b9dac865fd68a409efc17c682467b Apr 16 16:05:19.083124 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.083094 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5785d4fcdd-q2snw"] Apr 16 16:05:19.085329 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:05:19.085304 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf86cc5_0e80_47e9_b081_8510a5c57973.slice/crio-020e9ae6d16f6f6cab4d13ff57e7e95761f817f77310dcde9f67d4b8fb369236 WatchSource:0}: Error finding container 020e9ae6d16f6f6cab4d13ff57e7e95761f817f77310dcde9f67d4b8fb369236: Status 404 returned error can't find the container with id 020e9ae6d16f6f6cab4d13ff57e7e95761f817f77310dcde9f67d4b8fb369236 Apr 16 16:05:19.321989 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.321907 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:19.322179 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:19.322038 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:19.322179 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:19.322099 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls podName:7d332411-d30c-4247-a333-0a0f77355443 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:20.322082892 +0000 UTC m=+158.580829700 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-w6kz2" (UID: "7d332411-d30c-4247-a333-0a0f77355443") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:19.422843 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.422790 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:19.422996 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.422886 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:19.422996 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.422939 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls\") pod \"cluster-samples-operator-667775844f-28sdj\" (UID: \"25e38cb7-4445-4ec1-8afb-75356dc1c085\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:19.422996 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:19.422948 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 16 16:05:19.422996 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.422963 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:19.423259 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:19.423045 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert podName:5e3c6810-602d-4451-a18d-2b7f8b3e6245 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:20.423005194 +0000 UTC m=+158.681751986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert") pod "networking-console-plugin-5cb6cf4cb4-j6j9c" (UID: "5e3c6810-602d-4451-a18d-2b7f8b3e6245") : secret "networking-console-plugin-cert" not found Apr 16 16:05:19.423259 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:19.423088 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 16 16:05:19.423259 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:19.423107 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle podName:66494738-0f0c-42ff-b8e6-c0961cc5f3c1 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:20.423087446 +0000 UTC m=+158.681834253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle") pod "router-default-6fdcb8c7b8-jprpz" (UID: "66494738-0f0c-42ff-b8e6-c0961cc5f3c1") : configmap references non-existent config key: service-ca.crt Apr 16 16:05:19.423259 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:19.423136 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls podName:25e38cb7-4445-4ec1-8afb-75356dc1c085 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:20.423122826 +0000 UTC m=+158.681869628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls") pod "cluster-samples-operator-667775844f-28sdj" (UID: "25e38cb7-4445-4ec1-8afb-75356dc1c085") : secret "samples-operator-tls" not found Apr 16 16:05:19.423259 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:19.423088 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 16:05:19.423259 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:19.423171 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs podName:66494738-0f0c-42ff-b8e6-c0961cc5f3c1 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:20.423161702 +0000 UTC m=+158.681908521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs") pod "router-default-6fdcb8c7b8-jprpz" (UID: "66494738-0f0c-42ff-b8e6-c0961cc5f3c1") : secret "router-metrics-certs-default" not found Apr 16 16:05:19.800485 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.800432 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" event={"ID":"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a","Type":"ContainerStarted","Data":"61913e32e1407ad649441688d4af385e833b9dac865fd68a409efc17c682467b"} Apr 16 16:05:19.802947 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.802889 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-7d955d5dd4-k947k" event={"ID":"0c553c40-80b0-4906-9c86-e744ca5c2eba","Type":"ContainerStarted","Data":"6de8902033d0be4e9eac0c56b100e7f918d4f446277d55d820927d35355eb03a"} Apr 16 16:05:19.804308 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.804274 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" event={"ID":"dbf86cc5-0e80-47e9-b081-8510a5c57973","Type":"ContainerStarted","Data":"020e9ae6d16f6f6cab4d13ff57e7e95761f817f77310dcde9f67d4b8fb369236"} Apr 16 16:05:19.805226 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:19.805176 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" event={"ID":"9541790d-c119-472d-ab6a-4915d4065240","Type":"ContainerStarted","Data":"1f0a7dcd8034fc1a3bb87a9c912e9427640b2019784941764d6fc0078ab1413f"} Apr 16 16:05:20.336399 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:20.336359 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:20.336592 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:20.336520 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:20.336666 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:20.336597 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls podName:7d332411-d30c-4247-a333-0a0f77355443 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:22.336576722 +0000 UTC m=+160.595323516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-w6kz2" (UID: "7d332411-d30c-4247-a333-0a0f77355443") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:20.429228 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:20.429195 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-fpztb" podUID="cd5f8939-2604-4ca4-8d1e-05dc7f86fa41" Apr 16 16:05:20.437526 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:20.437490 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:20.437677 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:20.437597 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls\") pod \"cluster-samples-operator-667775844f-28sdj\" (UID: \"25e38cb7-4445-4ec1-8afb-75356dc1c085\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:20.437677 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:20.437633 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:20.437677 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:20.437659 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle podName:66494738-0f0c-42ff-b8e6-c0961cc5f3c1 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:22.437636118 +0000 UTC m=+160.696382955 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle") pod "router-default-6fdcb8c7b8-jprpz" (UID: "66494738-0f0c-42ff-b8e6-c0961cc5f3c1") : configmap references non-existent config key: service-ca.crt Apr 16 16:05:20.437885 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:20.437726 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:20.437885 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:20.437745 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 16:05:20.437885 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:20.437797 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs podName:66494738-0f0c-42ff-b8e6-c0961cc5f3c1 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:22.437780141 +0000 UTC m=+160.696526946 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs") pod "router-default-6fdcb8c7b8-jprpz" (UID: "66494738-0f0c-42ff-b8e6-c0961cc5f3c1") : secret "router-metrics-certs-default" not found Apr 16 16:05:20.437885 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:20.437841 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 16 16:05:20.437885 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:20.437883 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert podName:5e3c6810-602d-4451-a18d-2b7f8b3e6245 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:22.43787074 +0000 UTC m=+160.696617528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert") pod "networking-console-plugin-5cb6cf4cb4-j6j9c" (UID: "5e3c6810-602d-4451-a18d-2b7f8b3e6245") : secret "networking-console-plugin-cert" not found Apr 16 16:05:20.438149 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:20.437841 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 16 16:05:20.438149 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:20.437925 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls podName:25e38cb7-4445-4ec1-8afb-75356dc1c085 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:22.437914937 +0000 UTC m=+160.696661728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls") pod "cluster-samples-operator-667775844f-28sdj" (UID: "25e38cb7-4445-4ec1-8afb-75356dc1c085") : secret "samples-operator-tls" not found Apr 16 16:05:22.352158 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.352114 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:22.352670 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:22.352274 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:22.352670 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:22.352353 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls podName:7d332411-d30c-4247-a333-0a0f77355443 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:26.352334264 +0000 UTC m=+164.611081072 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-w6kz2" (UID: "7d332411-d30c-4247-a333-0a0f77355443") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:22.452750 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.452719 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:22.452909 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.452781 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls\") pod \"cluster-samples-operator-667775844f-28sdj\" (UID: \"25e38cb7-4445-4ec1-8afb-75356dc1c085\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:22.452909 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.452809 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:22.452909 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.452838 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:22.452909 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:22.452865 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle podName:66494738-0f0c-42ff-b8e6-c0961cc5f3c1 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:26.452846945 +0000 UTC m=+164.711593733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle") pod "router-default-6fdcb8c7b8-jprpz" (UID: "66494738-0f0c-42ff-b8e6-c0961cc5f3c1") : configmap references non-existent config key: service-ca.crt Apr 16 16:05:22.453146 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:22.452920 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 16 16:05:22.453146 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:22.452923 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 16:05:22.453146 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:22.452920 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 16 16:05:22.453146 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:22.452969 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert podName:5e3c6810-602d-4451-a18d-2b7f8b3e6245 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:26.452954794 +0000 UTC m=+164.711701611 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert") pod "networking-console-plugin-5cb6cf4cb4-j6j9c" (UID: "5e3c6810-602d-4451-a18d-2b7f8b3e6245") : secret "networking-console-plugin-cert" not found Apr 16 16:05:22.453146 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:22.452985 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls podName:25e38cb7-4445-4ec1-8afb-75356dc1c085 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:26.45297502 +0000 UTC m=+164.711721809 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls") pod "cluster-samples-operator-667775844f-28sdj" (UID: "25e38cb7-4445-4ec1-8afb-75356dc1c085") : secret "samples-operator-tls" not found Apr 16 16:05:22.453146 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:22.452998 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs podName:66494738-0f0c-42ff-b8e6-c0961cc5f3c1 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:26.452992504 +0000 UTC m=+164.711739292 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs") pod "router-default-6fdcb8c7b8-jprpz" (UID: "66494738-0f0c-42ff-b8e6-c0961cc5f3c1") : secret "router-metrics-certs-default" not found Apr 16 16:05:22.813833 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.813795 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" event={"ID":"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a","Type":"ContainerStarted","Data":"cd0358402bc9a7927251595612054680bf395a04e531871ce659d93ec57ac4fa"} Apr 16 16:05:22.815193 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.815169 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-7d955d5dd4-k947k" event={"ID":"0c553c40-80b0-4906-9c86-e744ca5c2eba","Type":"ContainerStarted","Data":"71233dff44f309aa330e8c84f642f9f2b3942a06411d955ac9224582c7e7fa18"} Apr 16 16:05:22.816588 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.816563 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" event={"ID":"dbf86cc5-0e80-47e9-b081-8510a5c57973","Type":"ContainerStarted","Data":"7bc12f2c8168f6f43583aa83edaff3e8925cbc1f1c7c1d326af80d16ac2b1416"} Apr 16 16:05:22.818031 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.817994 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/0.log" Apr 16 16:05:22.818134 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.818050 2576 generic.go:358] "Generic (PLEG): container finished" podID="9541790d-c119-472d-ab6a-4915d4065240" containerID="ba741176f6ca9ac3649d1e23277a9f7552be82d098b852638f5b1073c751c85a" exitCode=255 Apr 16 16:05:22.818134 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.818109 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" event={"ID":"9541790d-c119-472d-ab6a-4915d4065240","Type":"ContainerDied","Data":"ba741176f6ca9ac3649d1e23277a9f7552be82d098b852638f5b1073c751c85a"} Apr 16 16:05:22.818304 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.818288 2576 scope.go:117] "RemoveContainer" containerID="ba741176f6ca9ac3649d1e23277a9f7552be82d098b852638f5b1073c751c85a" Apr 16 16:05:22.832878 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.832834 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" podStartSLOduration=2.067436166 podStartE2EDuration="4.832820136s" podCreationTimestamp="2026-04-16 16:05:18 +0000 UTC" firstStartedPulling="2026-04-16 16:05:19.072870726 +0000 UTC m=+157.331617518" lastFinishedPulling="2026-04-16 16:05:21.838254685 +0000 UTC m=+160.097001488" observedRunningTime="2026-04-16 16:05:22.832444806 +0000 UTC m=+161.091191617" watchObservedRunningTime="2026-04-16 16:05:22.832820136 +0000 UTC m=+161.091567008" Apr 16 16:05:22.854381 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.854314 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-7d955d5dd4-k947k" podStartSLOduration=2.074386274 podStartE2EDuration="4.854294013s" podCreationTimestamp="2026-04-16 16:05:18 +0000 UTC" firstStartedPulling="2026-04-16 16:05:19.054193336 +0000 UTC m=+157.312940140" lastFinishedPulling="2026-04-16 16:05:21.834101077 +0000 UTC m=+160.092847879" observedRunningTime="2026-04-16 16:05:22.853065785 +0000 UTC m=+161.111812596" watchObservedRunningTime="2026-04-16 16:05:22.854294013 +0000 UTC m=+161.113040823" Apr 16 16:05:22.883571 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:22.883517 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" podStartSLOduration=2.131099588 podStartE2EDuration="4.88349639s" podCreationTimestamp="2026-04-16 16:05:18 +0000 UTC" firstStartedPulling="2026-04-16 16:05:19.087105673 +0000 UTC m=+157.345852462" lastFinishedPulling="2026-04-16 16:05:21.839502461 +0000 UTC m=+160.098249264" observedRunningTime="2026-04-16 16:05:22.883224221 +0000 UTC m=+161.141971031" watchObservedRunningTime="2026-04-16 16:05:22.88349639 +0000 UTC m=+161.142243201" Apr 16 16:05:23.562899 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:23.562863 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls\") pod \"image-registry-58d7958c46-l2zxs\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:05:23.563375 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:23.562924 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:05:23.563375 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:23.562950 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:05:23.563375 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:23.563054 2576 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found Apr 16 16:05:23.563375 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:23.563058 2576 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Apr 16 16:05:23.563375 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:23.563083 2576 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-58d7958c46-l2zxs: secret "image-registry-tls" not found Apr 16 16:05:23.563375 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:23.563110 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Apr 16 16:05:23.563375 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:23.563115 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert podName:f4bb9693-c1c3-4ecc-b82d-2ea9913caa85 nodeName:}" failed. No retries permitted until 2026-04-16 16:07:25.563096729 +0000 UTC m=+283.821843536 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert") pod "ingress-canary-kmvkl" (UID: "f4bb9693-c1c3-4ecc-b82d-2ea9913caa85") : secret "canary-serving-cert" not found Apr 16 16:05:23.563375 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:23.563261 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls podName:7ed80516-2524-487c-8c72-d7ded9a82159 nodeName:}" failed. No retries permitted until 2026-04-16 16:07:25.563243748 +0000 UTC m=+283.821990535 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls") pod "image-registry-58d7958c46-l2zxs" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159") : secret "image-registry-tls" not found Apr 16 16:05:23.563375 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:23.563286 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls podName:ad7697a0-8cda-49df-861c-5762c30a7794 nodeName:}" failed. No retries permitted until 2026-04-16 16:07:25.563277052 +0000 UTC m=+283.822023847 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls") pod "dns-default-mt4rq" (UID: "ad7697a0-8cda-49df-861c-5762c30a7794") : secret "dns-default-metrics-tls" not found Apr 16 16:05:23.821687 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:23.821613 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/1.log" Apr 16 16:05:23.822026 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:23.821995 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/0.log" Apr 16 16:05:23.822079 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:23.822043 2576 generic.go:358] "Generic (PLEG): container finished" podID="9541790d-c119-472d-ab6a-4915d4065240" containerID="fc5cb6ce9058888a2e6293566f8e2715f127d03e0ec39a7f1773ba89c0bb5c6a" exitCode=255 Apr 16 16:05:23.822117 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:23.822074 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" event={"ID":"9541790d-c119-472d-ab6a-4915d4065240","Type":"ContainerDied","Data":"fc5cb6ce9058888a2e6293566f8e2715f127d03e0ec39a7f1773ba89c0bb5c6a"} Apr 16 16:05:23.822150 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:23.822129 2576 scope.go:117] "RemoveContainer" containerID="ba741176f6ca9ac3649d1e23277a9f7552be82d098b852638f5b1073c751c85a" Apr 16 16:05:23.822384 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:23.822363 2576 scope.go:117] "RemoveContainer" containerID="fc5cb6ce9058888a2e6293566f8e2715f127d03e0ec39a7f1773ba89c0bb5c6a" Apr 16 16:05:23.822627 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:23.822600 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-d87b8d5fc-qb2g6_openshift-console-operator(9541790d-c119-472d-ab6a-4915d4065240)\"" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" podUID="9541790d-c119-472d-ab6a-4915d4065240" Apr 16 16:05:24.268908 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.268874 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-bfc587fb7-x2srs"] Apr 16 16:05:24.273099 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.273079 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" Apr 16 16:05:24.275876 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.275859 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Apr 16 16:05:24.276266 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.276248 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Apr 16 16:05:24.276346 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.276248 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Apr 16 16:05:24.276426 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.276408 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Apr 16 16:05:24.276908 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.276895 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9kfs2\"" Apr 16 16:05:24.284602 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.284583 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-bfc587fb7-x2srs"] Apr 16 16:05:24.371759 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.371729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96kds\" (UniqueName: \"kubernetes.io/projected/b57e8f88-ec1f-475c-93fb-2efc32bbff4d-kube-api-access-96kds\") pod \"service-ca-bfc587fb7-x2srs\" (UID: \"b57e8f88-ec1f-475c-93fb-2efc32bbff4d\") " pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" Apr 16 16:05:24.371759 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.371763 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b57e8f88-ec1f-475c-93fb-2efc32bbff4d-signing-key\") pod \"service-ca-bfc587fb7-x2srs\" (UID: \"b57e8f88-ec1f-475c-93fb-2efc32bbff4d\") " pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" Apr 16 16:05:24.371979 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.371784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b57e8f88-ec1f-475c-93fb-2efc32bbff4d-signing-cabundle\") pod \"service-ca-bfc587fb7-x2srs\" (UID: \"b57e8f88-ec1f-475c-93fb-2efc32bbff4d\") " pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" Apr 16 16:05:24.472334 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.472300 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-96kds\" (UniqueName: \"kubernetes.io/projected/b57e8f88-ec1f-475c-93fb-2efc32bbff4d-kube-api-access-96kds\") pod \"service-ca-bfc587fb7-x2srs\" (UID: \"b57e8f88-ec1f-475c-93fb-2efc32bbff4d\") " pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" Apr 16 16:05:24.472334 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.472339 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b57e8f88-ec1f-475c-93fb-2efc32bbff4d-signing-key\") pod \"service-ca-bfc587fb7-x2srs\" (UID: \"b57e8f88-ec1f-475c-93fb-2efc32bbff4d\") " pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" Apr 16 16:05:24.472531 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.472356 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b57e8f88-ec1f-475c-93fb-2efc32bbff4d-signing-cabundle\") pod \"service-ca-bfc587fb7-x2srs\" (UID: \"b57e8f88-ec1f-475c-93fb-2efc32bbff4d\") " pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" Apr 16 16:05:24.473063 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.473038 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b57e8f88-ec1f-475c-93fb-2efc32bbff4d-signing-cabundle\") pod \"service-ca-bfc587fb7-x2srs\" (UID: \"b57e8f88-ec1f-475c-93fb-2efc32bbff4d\") " pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" Apr 16 16:05:24.474670 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.474644 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b57e8f88-ec1f-475c-93fb-2efc32bbff4d-signing-key\") pod \"service-ca-bfc587fb7-x2srs\" (UID: \"b57e8f88-ec1f-475c-93fb-2efc32bbff4d\") " pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" Apr 16 16:05:24.482639 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.482619 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96kds\" (UniqueName: \"kubernetes.io/projected/b57e8f88-ec1f-475c-93fb-2efc32bbff4d-kube-api-access-96kds\") pod \"service-ca-bfc587fb7-x2srs\" (UID: \"b57e8f88-ec1f-475c-93fb-2efc32bbff4d\") " pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" Apr 16 16:05:24.581627 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.581546 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" Apr 16 16:05:24.694402 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.694364 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-bfc587fb7-x2srs"] Apr 16 16:05:24.697623 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:05:24.697598 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57e8f88_ec1f_475c_93fb_2efc32bbff4d.slice/crio-439a0c48bd11fa79998a54f4d5855c465206b444ac34271d4cd0cf411a9605a1 WatchSource:0}: Error finding container 439a0c48bd11fa79998a54f4d5855c465206b444ac34271d4cd0cf411a9605a1: Status 404 returned error can't find the container with id 439a0c48bd11fa79998a54f4d5855c465206b444ac34271d4cd0cf411a9605a1 Apr 16 16:05:24.828160 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.828135 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/1.log" Apr 16 16:05:24.828534 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.828512 2576 scope.go:117] "RemoveContainer" containerID="fc5cb6ce9058888a2e6293566f8e2715f127d03e0ec39a7f1773ba89c0bb5c6a" Apr 16 16:05:24.828724 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:24.828707 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-d87b8d5fc-qb2g6_openshift-console-operator(9541790d-c119-472d-ab6a-4915d4065240)\"" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" podUID="9541790d-c119-472d-ab6a-4915d4065240" Apr 16 16:05:24.829139 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:24.829119 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" event={"ID":"b57e8f88-ec1f-475c-93fb-2efc32bbff4d","Type":"ContainerStarted","Data":"439a0c48bd11fa79998a54f4d5855c465206b444ac34271d4cd0cf411a9605a1"} Apr 16 16:05:25.075314 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:25.075287 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-klzvx_dd45d1df-4307-46b1-878a-6073d5e56455/dns-node-resolver/0.log" Apr 16 16:05:25.875355 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:25.875325 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bgdhp_0020ce4c-790d-41fd-9e3a-6ada38675cda/node-ca/0.log" Apr 16 16:05:26.387869 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:26.387831 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:26.388073 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:26.387969 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:26.388073 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:26.388055 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls podName:7d332411-d30c-4247-a333-0a0f77355443 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:34.388036713 +0000 UTC m=+172.646783515 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-w6kz2" (UID: "7d332411-d30c-4247-a333-0a0f77355443") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:26.488445 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:26.488367 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:26.488445 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:26.488434 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls\") pod \"cluster-samples-operator-667775844f-28sdj\" (UID: \"25e38cb7-4445-4ec1-8afb-75356dc1c085\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:26.488657 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:26.488518 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Apr 16 16:05:26.488657 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:26.488559 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle podName:66494738-0f0c-42ff-b8e6-c0961cc5f3c1 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:34.488536902 +0000 UTC m=+172.747283707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle") pod "router-default-6fdcb8c7b8-jprpz" (UID: "66494738-0f0c-42ff-b8e6-c0961cc5f3c1") : configmap references non-existent config key: service-ca.crt Apr 16 16:05:26.488657 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:26.488586 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls podName:25e38cb7-4445-4ec1-8afb-75356dc1c085 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:34.488577193 +0000 UTC m=+172.747324006 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls") pod "cluster-samples-operator-667775844f-28sdj" (UID: "25e38cb7-4445-4ec1-8afb-75356dc1c085") : secret "samples-operator-tls" not found Apr 16 16:05:26.488657 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:26.488610 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:26.488837 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:26.488681 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:26.488837 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:26.488757 2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Apr 16 16:05:26.488837 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:26.488800 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs podName:66494738-0f0c-42ff-b8e6-c0961cc5f3c1 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:34.488787759 +0000 UTC m=+172.747534560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs") pod "router-default-6fdcb8c7b8-jprpz" (UID: "66494738-0f0c-42ff-b8e6-c0961cc5f3c1") : secret "router-metrics-certs-default" not found Apr 16 16:05:26.488837 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:26.488811 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 16 16:05:26.488966 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:26.488855 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert podName:5e3c6810-602d-4451-a18d-2b7f8b3e6245 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:34.488841194 +0000 UTC m=+172.747588002 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert") pod "networking-console-plugin-5cb6cf4cb4-j6j9c" (UID: "5e3c6810-602d-4451-a18d-2b7f8b3e6245") : secret "networking-console-plugin-cert" not found Apr 16 16:05:26.837915 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:26.837830 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" event={"ID":"b57e8f88-ec1f-475c-93fb-2efc32bbff4d","Type":"ContainerStarted","Data":"b2f56b2e583fd3f11265e44a177bca7de8f1c8b2465abdc3b15d294c55d5f69a"} Apr 16 16:05:26.864148 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:26.864095 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca/service-ca-bfc587fb7-x2srs" podStartSLOduration=1.330319475 podStartE2EDuration="2.864079689s" podCreationTimestamp="2026-04-16 16:05:24 +0000 UTC" firstStartedPulling="2026-04-16 16:05:24.699487216 +0000 UTC m=+162.958234007" lastFinishedPulling="2026-04-16 16:05:26.233247422 +0000 UTC m=+164.491994221" observedRunningTime="2026-04-16 16:05:26.863293715 +0000 UTC m=+165.122040524" watchObservedRunningTime="2026-04-16 16:05:26.864079689 +0000 UTC m=+165.122826499" Apr 16 16:05:28.817409 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:28.817372 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:28.817409 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:28.817418 2576 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:28.817873 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:28.817859 2576 scope.go:117] "RemoveContainer" containerID="fc5cb6ce9058888a2e6293566f8e2715f127d03e0ec39a7f1773ba89c0bb5c6a" Apr 16 16:05:28.818100 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:28.818080 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-d87b8d5fc-qb2g6_openshift-console-operator(9541790d-c119-472d-ab6a-4915d4065240)\"" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" podUID="9541790d-c119-472d-ab6a-4915d4065240" Apr 16 16:05:31.414190 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:31.414149 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:05:34.462630 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:34.462583 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:34.463046 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:34.462730 2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:34.463046 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:34.462797 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls podName:7d332411-d30c-4247-a333-0a0f77355443 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:50.462782228 +0000 UTC m=+188.721529016 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6667474d89-w6kz2" (UID: "7d332411-d30c-4247-a333-0a0f77355443") : secret "cluster-monitoring-operator-tls" not found Apr 16 16:05:34.564117 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:34.564069 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:34.564309 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:34.564174 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls\") pod \"cluster-samples-operator-667775844f-28sdj\" (UID: \"25e38cb7-4445-4ec1-8afb-75356dc1c085\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:34.564309 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:34.564206 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:34.564309 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:34.564248 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle podName:66494738-0f0c-42ff-b8e6-c0961cc5f3c1 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:50.564227959 +0000 UTC m=+188.822974751 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle") pod "router-default-6fdcb8c7b8-jprpz" (UID: "66494738-0f0c-42ff-b8e6-c0961cc5f3c1") : configmap references non-existent config key: service-ca.crt Apr 16 16:05:34.564477 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:34.564309 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:34.564528 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:34.564476 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Apr 16 16:05:34.564528 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:34.564521 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert podName:5e3c6810-602d-4451-a18d-2b7f8b3e6245 nodeName:}" failed. No retries permitted until 2026-04-16 16:05:50.564508258 +0000 UTC m=+188.823255071 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert") pod "networking-console-plugin-5cb6cf4cb4-j6j9c" (UID: "5e3c6810-602d-4451-a18d-2b7f8b3e6245") : secret "networking-console-plugin-cert" not found Apr 16 16:05:34.566957 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:34.566930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-metrics-certs\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:34.567121 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:34.567039 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/25e38cb7-4445-4ec1-8afb-75356dc1c085-samples-operator-tls\") pod \"cluster-samples-operator-667775844f-28sdj\" (UID: \"25e38cb7-4445-4ec1-8afb-75356dc1c085\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:34.627562 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:34.627526 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" Apr 16 16:05:34.758786 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:34.758704 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj"] Apr 16 16:05:34.861229 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:34.861198 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" event={"ID":"25e38cb7-4445-4ec1-8afb-75356dc1c085","Type":"ContainerStarted","Data":"9551e1b780569958e09691f49ba70dc10b280ca116c52943f0e59dd522db33d9"} Apr 16 16:05:36.872361 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:36.872328 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" event={"ID":"25e38cb7-4445-4ec1-8afb-75356dc1c085","Type":"ContainerStarted","Data":"6b0fc7ccbec5547d07f6e0098f758396f40f1fcf682a3c1750407438a68e09a5"} Apr 16 16:05:36.872361 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:36.872362 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" event={"ID":"25e38cb7-4445-4ec1-8afb-75356dc1c085","Type":"ContainerStarted","Data":"035de572bbbe702d8c822166cae01d0dc552740ca91bfb9c53ef156a7e753bca"} Apr 16 16:05:36.892879 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:36.892833 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-667775844f-28sdj" podStartSLOduration=17.505381811 podStartE2EDuration="18.892819578s" podCreationTimestamp="2026-04-16 16:05:18 +0000 UTC" firstStartedPulling="2026-04-16 16:05:34.807790025 +0000 UTC m=+173.066536826" lastFinishedPulling="2026-04-16 16:05:36.195227804 +0000 UTC m=+174.453974593" observedRunningTime="2026-04-16 16:05:36.89136174 +0000 UTC m=+175.150108550" watchObservedRunningTime="2026-04-16 16:05:36.892819578 +0000 UTC m=+175.151566387" Apr 16 16:05:41.414448 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:41.414415 2576 scope.go:117] "RemoveContainer" containerID="fc5cb6ce9058888a2e6293566f8e2715f127d03e0ec39a7f1773ba89c0bb5c6a" Apr 16 16:05:41.886692 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:41.886662 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:05:41.887060 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:41.887043 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/1.log" Apr 16 16:05:41.887115 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:41.887077 2576 generic.go:358] "Generic (PLEG): container finished" podID="9541790d-c119-472d-ab6a-4915d4065240" containerID="be5abf022a642338fab76b36d8765ba74f40da64a13a069ae256f9e9f9274912" exitCode=255 Apr 16 16:05:41.887169 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:41.887123 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" event={"ID":"9541790d-c119-472d-ab6a-4915d4065240","Type":"ContainerDied","Data":"be5abf022a642338fab76b36d8765ba74f40da64a13a069ae256f9e9f9274912"} Apr 16 16:05:41.887169 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:41.887156 2576 scope.go:117] "RemoveContainer" containerID="fc5cb6ce9058888a2e6293566f8e2715f127d03e0ec39a7f1773ba89c0bb5c6a" Apr 16 16:05:41.887467 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:41.887443 2576 scope.go:117] "RemoveContainer" containerID="be5abf022a642338fab76b36d8765ba74f40da64a13a069ae256f9e9f9274912" Apr 16 16:05:41.887672 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:41.887651 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-d87b8d5fc-qb2g6_openshift-console-operator(9541790d-c119-472d-ab6a-4915d4065240)\"" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" podUID="9541790d-c119-472d-ab6a-4915d4065240" Apr 16 16:05:42.890942 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:42.890915 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:05:43.991211 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:43.991180 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gpd5d"] Apr 16 16:05:43.995702 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:43.995682 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.000451 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.000431 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Apr 16 16:05:44.000451 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.000443 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8s6r8\"" Apr 16 16:05:44.000588 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.000432 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Apr 16 16:05:44.008668 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.008645 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gpd5d"] Apr 16 16:05:44.141836 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.141808 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17a67b97-ec02-466c-9534-4b23a9e0997e-data-volume\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.141998 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.141844 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17a67b97-ec02-466c-9534-4b23a9e0997e-crio-socket\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.141998 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.141876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a67b97-ec02-466c-9534-4b23a9e0997e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.141998 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.141896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17a67b97-ec02-466c-9534-4b23a9e0997e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.141998 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.141946 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z5dn\" (UniqueName: \"kubernetes.io/projected/17a67b97-ec02-466c-9534-4b23a9e0997e-kube-api-access-5z5dn\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.242914 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.242834 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5z5dn\" (UniqueName: \"kubernetes.io/projected/17a67b97-ec02-466c-9534-4b23a9e0997e-kube-api-access-5z5dn\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.243077 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.242963 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17a67b97-ec02-466c-9534-4b23a9e0997e-data-volume\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.243077 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.242990 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17a67b97-ec02-466c-9534-4b23a9e0997e-crio-socket\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.243077 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.243055 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a67b97-ec02-466c-9534-4b23a9e0997e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.243249 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.243080 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17a67b97-ec02-466c-9534-4b23a9e0997e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.243249 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.243198 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/17a67b97-ec02-466c-9534-4b23a9e0997e-crio-socket\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.243380 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.243363 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/17a67b97-ec02-466c-9534-4b23a9e0997e-data-volume\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.243676 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.243654 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/17a67b97-ec02-466c-9534-4b23a9e0997e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.245443 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.245424 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/17a67b97-ec02-466c-9534-4b23a9e0997e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.255791 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.255764 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z5dn\" (UniqueName: \"kubernetes.io/projected/17a67b97-ec02-466c-9534-4b23a9e0997e-kube-api-access-5z5dn\") pod \"insights-runtime-extractor-gpd5d\" (UID: \"17a67b97-ec02-466c-9534-4b23a9e0997e\") " pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.304003 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.303976 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gpd5d" Apr 16 16:05:44.457875 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.457842 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gpd5d"] Apr 16 16:05:44.459857 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:05:44.459830 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17a67b97_ec02_466c_9534_4b23a9e0997e.slice/crio-8fc14c6b7b40e7c715b4f689f3c93a526b115528f3dda37ea76d1c85f2a79543 WatchSource:0}: Error finding container 8fc14c6b7b40e7c715b4f689f3c93a526b115528f3dda37ea76d1c85f2a79543: Status 404 returned error can't find the container with id 8fc14c6b7b40e7c715b4f689f3c93a526b115528f3dda37ea76d1c85f2a79543 Apr 16 16:05:44.897058 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.896963 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpd5d" event={"ID":"17a67b97-ec02-466c-9534-4b23a9e0997e","Type":"ContainerStarted","Data":"63d12904f3d06a05a47b848529828e4468b2e64ce7b21e0103d78b704dc15874"} Apr 16 16:05:44.897058 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:44.896999 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpd5d" event={"ID":"17a67b97-ec02-466c-9534-4b23a9e0997e","Type":"ContainerStarted","Data":"8fc14c6b7b40e7c715b4f689f3c93a526b115528f3dda37ea76d1c85f2a79543"} Apr 16 16:05:45.901279 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:45.901243 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpd5d" event={"ID":"17a67b97-ec02-466c-9534-4b23a9e0997e","Type":"ContainerStarted","Data":"92d730fd8b29a0ed76e221e11ef6e1c3e414a11c5628cdf829baa8c646d31247"} Apr 16 16:05:46.909480 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:46.909440 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gpd5d" event={"ID":"17a67b97-ec02-466c-9534-4b23a9e0997e","Type":"ContainerStarted","Data":"55f835abfc7d105ff66bd2ac0142dd0b5474f11e6e8014a9abd0e2415ab1192c"} Apr 16 16:05:48.817443 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:48.817412 2576 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:48.817443 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:48.817452 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:05:48.817841 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:48.817804 2576 scope.go:117] "RemoveContainer" containerID="be5abf022a642338fab76b36d8765ba74f40da64a13a069ae256f9e9f9274912" Apr 16 16:05:48.817993 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:05:48.817976 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-d87b8d5fc-qb2g6_openshift-console-operator(9541790d-c119-472d-ab6a-4915d4065240)\"" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" podUID="9541790d-c119-472d-ab6a-4915d4065240" Apr 16 16:05:48.839984 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:48.839945 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gpd5d" podStartSLOduration=3.675974647 podStartE2EDuration="5.839930248s" podCreationTimestamp="2026-04-16 16:05:43 +0000 UTC" firstStartedPulling="2026-04-16 16:05:44.514305253 +0000 UTC m=+182.773052056" lastFinishedPulling="2026-04-16 16:05:46.678260869 +0000 UTC m=+184.937007657" observedRunningTime="2026-04-16 16:05:46.95608287 +0000 UTC m=+185.214829679" watchObservedRunningTime="2026-04-16 16:05:48.839930248 +0000 UTC m=+187.098677052" Apr 16 16:05:50.495467 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.495424 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:50.497837 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.497819 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d332411-d30c-4247-a333-0a0f77355443-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6667474d89-w6kz2\" (UID: \"7d332411-d30c-4247-a333-0a0f77355443\") " pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:50.596384 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.596343 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:50.596602 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.596413 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:50.596940 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.596920 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66494738-0f0c-42ff-b8e6-c0961cc5f3c1-service-ca-bundle\") pod \"router-default-6fdcb8c7b8-jprpz\" (UID: \"66494738-0f0c-42ff-b8e6-c0961cc5f3c1\") " pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:50.598742 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.598718 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5e3c6810-602d-4451-a18d-2b7f8b3e6245-networking-console-plugin-cert\") pod \"networking-console-plugin-5cb6cf4cb4-j6j9c\" (UID: \"5e3c6810-602d-4451-a18d-2b7f8b3e6245\") " pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:50.732815 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.732777 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pblvm\"" Apr 16 16:05:50.741082 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.741059 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" Apr 16 16:05:50.817303 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.817277 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-74jmg\"" Apr 16 16:05:50.824574 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.824550 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" Apr 16 16:05:50.836026 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.835981 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-d7b7v\"" Apr 16 16:05:50.844554 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.844529 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:50.861637 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.861608 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2"] Apr 16 16:05:50.864449 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:05:50.864412 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d332411_d30c_4247_a333_0a0f77355443.slice/crio-07004fc224952962f3e53fdd6d913b5bb1dea8b465fd6c8ac26b80278741cb39 WatchSource:0}: Error finding container 07004fc224952962f3e53fdd6d913b5bb1dea8b465fd6c8ac26b80278741cb39: Status 404 returned error can't find the container with id 07004fc224952962f3e53fdd6d913b5bb1dea8b465fd6c8ac26b80278741cb39 Apr 16 16:05:50.921089 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.921051 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" event={"ID":"7d332411-d30c-4247-a333-0a0f77355443","Type":"ContainerStarted","Data":"07004fc224952962f3e53fdd6d913b5bb1dea8b465fd6c8ac26b80278741cb39"} Apr 16 16:05:50.958237 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.958217 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c"] Apr 16 16:05:50.960583 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:05:50.960559 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e3c6810_602d_4451_a18d_2b7f8b3e6245.slice/crio-da15224c3d0226032b029dee5caf9a30dd9cecf30bdc44ccf19bece6f6b8b858 WatchSource:0}: Error finding container da15224c3d0226032b029dee5caf9a30dd9cecf30bdc44ccf19bece6f6b8b858: Status 404 returned error can't find the container with id da15224c3d0226032b029dee5caf9a30dd9cecf30bdc44ccf19bece6f6b8b858 Apr 16 16:05:50.982211 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:50.982187 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6fdcb8c7b8-jprpz"] Apr 16 16:05:50.984796 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:05:50.984770 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66494738_0f0c_42ff_b8e6_c0961cc5f3c1.slice/crio-a52797ed24ac9c64797f8d17b82552be41e27169794228c120074a1f60a9a5ed WatchSource:0}: Error finding container a52797ed24ac9c64797f8d17b82552be41e27169794228c120074a1f60a9a5ed: Status 404 returned error can't find the container with id a52797ed24ac9c64797f8d17b82552be41e27169794228c120074a1f60a9a5ed Apr 16 16:05:51.925742 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:51.925702 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" event={"ID":"66494738-0f0c-42ff-b8e6-c0961cc5f3c1","Type":"ContainerStarted","Data":"bd82eec9d41ad47189e76dd08154d2176ae7762ff8878b846d82987f9f9e32b8"} Apr 16 16:05:51.925742 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:51.925747 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" event={"ID":"66494738-0f0c-42ff-b8e6-c0961cc5f3c1","Type":"ContainerStarted","Data":"a52797ed24ac9c64797f8d17b82552be41e27169794228c120074a1f60a9a5ed"} Apr 16 16:05:51.926878 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:51.926853 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" event={"ID":"5e3c6810-602d-4451-a18d-2b7f8b3e6245","Type":"ContainerStarted","Data":"da15224c3d0226032b029dee5caf9a30dd9cecf30bdc44ccf19bece6f6b8b858"} Apr 16 16:05:52.439302 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:52.439241 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" podStartSLOduration=34.43922557 podStartE2EDuration="34.43922557s" podCreationTimestamp="2026-04-16 16:05:18 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:05:51.960850197 +0000 UTC m=+190.219597006" watchObservedRunningTime="2026-04-16 16:05:52.43922557 +0000 UTC m=+190.697972379" Apr 16 16:05:52.845708 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:52.845667 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:52.848241 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:52.848219 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:52.930890 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:52.930850 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" event={"ID":"5e3c6810-602d-4451-a18d-2b7f8b3e6245","Type":"ContainerStarted","Data":"ab4a3a2b3da7e814afc5dc7c2b08036227ddef3f8fdb907442d30106f4965f94"} Apr 16 16:05:52.932136 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:52.932108 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" event={"ID":"7d332411-d30c-4247-a333-0a0f77355443","Type":"ContainerStarted","Data":"31cfc482fe8190c205d6f90c9a52e829a904cf069c780a3375c49f6476482857"} Apr 16 16:05:52.932484 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:52.932467 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:52.933709 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:52.933689 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6fdcb8c7b8-jprpz" Apr 16 16:05:52.949333 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:52.949289 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5cb6cf4cb4-j6j9c" podStartSLOduration=33.17484794 podStartE2EDuration="34.949279102s" podCreationTimestamp="2026-04-16 16:05:18 +0000 UTC" firstStartedPulling="2026-04-16 16:05:50.962350879 +0000 UTC m=+189.221097671" lastFinishedPulling="2026-04-16 16:05:52.736782041 +0000 UTC m=+190.995528833" observedRunningTime="2026-04-16 16:05:52.948264449 +0000 UTC m=+191.207011260" watchObservedRunningTime="2026-04-16 16:05:52.949279102 +0000 UTC m=+191.208025946" Apr 16 16:05:52.967821 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:52.967781 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6667474d89-w6kz2" podStartSLOduration=33.097985589 podStartE2EDuration="34.967772764s" podCreationTimestamp="2026-04-16 16:05:18 +0000 UTC" firstStartedPulling="2026-04-16 16:05:50.866369713 +0000 UTC m=+189.125116500" lastFinishedPulling="2026-04-16 16:05:52.736156888 +0000 UTC m=+190.994903675" observedRunningTime="2026-04-16 16:05:52.967036734 +0000 UTC m=+191.225783545" watchObservedRunningTime="2026-04-16 16:05:52.967772764 +0000 UTC m=+191.226519574" Apr 16 16:05:53.322394 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:53.322359 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h"] Apr 16 16:05:53.326491 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:53.326472 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h" Apr 16 16:05:53.329422 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:53.329362 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Apr 16 16:05:53.329515 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:53.329469 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7wfsp\"" Apr 16 16:05:53.334926 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:53.334904 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h"] Apr 16 16:05:53.418471 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:53.418426 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2891f287-27d8-4b90-b731-016769f80008-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-nwx8h\" (UID: \"2891f287-27d8-4b90-b731-016769f80008\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h" Apr 16 16:05:53.519700 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:53.519664 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2891f287-27d8-4b90-b731-016769f80008-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-nwx8h\" (UID: \"2891f287-27d8-4b90-b731-016769f80008\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h" Apr 16 16:05:53.522114 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:53.522093 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2891f287-27d8-4b90-b731-016769f80008-tls-certificates\") pod \"prometheus-operator-admission-webhook-9cb97cd87-nwx8h\" (UID: \"2891f287-27d8-4b90-b731-016769f80008\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h" Apr 16 16:05:53.637561 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:53.637469 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h" Apr 16 16:05:53.753209 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:53.752998 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h"] Apr 16 16:05:53.755365 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:05:53.755337 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2891f287_27d8_4b90_b731_016769f80008.slice/crio-cbba91a735f2c36d2927c5fc8d4f81659d0b4568f4a2ee904465ef18a87e9096 WatchSource:0}: Error finding container cbba91a735f2c36d2927c5fc8d4f81659d0b4568f4a2ee904465ef18a87e9096: Status 404 returned error can't find the container with id cbba91a735f2c36d2927c5fc8d4f81659d0b4568f4a2ee904465ef18a87e9096 Apr 16 16:05:53.935696 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:53.935605 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h" event={"ID":"2891f287-27d8-4b90-b731-016769f80008","Type":"ContainerStarted","Data":"cbba91a735f2c36d2927c5fc8d4f81659d0b4568f4a2ee904465ef18a87e9096"} Apr 16 16:05:54.939067 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:54.938972 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h" event={"ID":"2891f287-27d8-4b90-b731-016769f80008","Type":"ContainerStarted","Data":"598c374b76c8c8b8d804c467bb3b7bdafecb484dbd587512417c025974a9028b"} Apr 16 16:05:54.939067 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:54.939037 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h" Apr 16 16:05:54.943731 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:54.943710 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h" Apr 16 16:05:54.958713 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:05:54.958671 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-9cb97cd87-nwx8h" podStartSLOduration=1.062152096 podStartE2EDuration="1.958659777s" podCreationTimestamp="2026-04-16 16:05:53 +0000 UTC" firstStartedPulling="2026-04-16 16:05:53.757355539 +0000 UTC m=+192.016102330" lastFinishedPulling="2026-04-16 16:05:54.653863221 +0000 UTC m=+192.912610011" observedRunningTime="2026-04-16 16:05:54.957322088 +0000 UTC m=+193.216069105" watchObservedRunningTime="2026-04-16 16:05:54.958659777 +0000 UTC m=+193.217406587" Apr 16 16:06:00.414975 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.414947 2576 scope.go:117] "RemoveContainer" containerID="be5abf022a642338fab76b36d8765ba74f40da64a13a069ae256f9e9f9274912" Apr 16 16:06:00.415382 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:06:00.415201 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-d87b8d5fc-qb2g6_openshift-console-operator(9541790d-c119-472d-ab6a-4915d4065240)\"" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" podUID="9541790d-c119-472d-ab6a-4915d4065240" Apr 16 16:06:00.842415 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.842380 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6fv2w"] Apr 16 16:06:00.846077 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.846053 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.850225 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.850190 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Apr 16 16:06:00.851382 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.851362 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Apr 16 16:06:00.851382 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.851374 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Apr 16 16:06:00.851717 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.851699 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nrtjc\"" Apr 16 16:06:00.851831 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.851815 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Apr 16 16:06:00.882467 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.882434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8756041d-afbd-4507-a82f-426bddd41acb-sys\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.882597 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.882520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-textfile\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.882597 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.882549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-wtmp\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.882597 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.882586 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8756041d-afbd-4507-a82f-426bddd41acb-metrics-client-ca\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.882758 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.882605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-tls\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.882758 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.882623 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8756041d-afbd-4507-a82f-426bddd41acb-root\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.882758 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.882649 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.882758 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.882726 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-accelerators-collector-config\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.882758 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.882753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4thp\" (UniqueName: \"kubernetes.io/projected/8756041d-afbd-4507-a82f-426bddd41acb-kube-api-access-f4thp\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.983866 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.983837 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-textfile\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984047 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.983883 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-wtmp\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984047 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.983933 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8756041d-afbd-4507-a82f-426bddd41acb-metrics-client-ca\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984047 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.983960 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-tls\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984213 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.984054 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8756041d-afbd-4507-a82f-426bddd41acb-root\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984213 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:06:00.984087 2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Apr 16 16:06:00.984213 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.984107 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-wtmp\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984213 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.984118 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984213 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:06:00.984145 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-tls podName:8756041d-afbd-4507-a82f-426bddd41acb nodeName:}" failed. No retries permitted until 2026-04-16 16:06:01.484127659 +0000 UTC m=+199.742874466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-tls") pod "node-exporter-6fv2w" (UID: "8756041d-afbd-4507-a82f-426bddd41acb") : secret "node-exporter-tls" not found Apr 16 16:06:00.984213 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.984162 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8756041d-afbd-4507-a82f-426bddd41acb-root\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984525 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.984230 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-accelerators-collector-config\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984525 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.984264 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f4thp\" (UniqueName: \"kubernetes.io/projected/8756041d-afbd-4507-a82f-426bddd41acb-kube-api-access-f4thp\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984525 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.984305 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8756041d-afbd-4507-a82f-426bddd41acb-sys\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984525 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.984234 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-textfile\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984525 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.984479 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8756041d-afbd-4507-a82f-426bddd41acb-sys\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.984777 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.984673 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-accelerators-collector-config\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.985168 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.985148 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8756041d-afbd-4507-a82f-426bddd41acb-metrics-client-ca\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.986541 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.986525 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:00.995896 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:00.995877 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4thp\" (UniqueName: \"kubernetes.io/projected/8756041d-afbd-4507-a82f-426bddd41acb-kube-api-access-f4thp\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:01.488509 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.488464 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-tls\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:01.490960 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.490938 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8756041d-afbd-4507-a82f-426bddd41acb-node-exporter-tls\") pod \"node-exporter-6fv2w\" (UID: \"8756041d-afbd-4507-a82f-426bddd41acb\") " pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:01.757396 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.757309 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6fv2w" Apr 16 16:06:01.765920 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:06:01.765892 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8756041d_afbd_4507_a82f_426bddd41acb.slice/crio-d8f653a39a55516f700325c7452cf5a6059d799f1b1476072a9be242f0b79f4c WatchSource:0}: Error finding container d8f653a39a55516f700325c7452cf5a6059d799f1b1476072a9be242f0b79f4c: Status 404 returned error can't find the container with id d8f653a39a55516f700325c7452cf5a6059d799f1b1476072a9be242f0b79f4c Apr 16 16:06:01.880268 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.880232 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:06:01.888755 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.888723 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.893979 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.893946 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Apr 16 16:06:01.893979 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.893945 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Apr 16 16:06:01.893979 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.893979 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Apr 16 16:06:01.894242 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.893945 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Apr 16 16:06:01.894242 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.893945 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Apr 16 16:06:01.894242 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.894135 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Apr 16 16:06:01.894395 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.894363 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Apr 16 16:06:01.894449 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.894432 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Apr 16 16:06:01.895288 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.895249 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Apr 16 16:06:01.895410 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.895251 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rzvwb\"" Apr 16 16:06:01.903076 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.903045 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:06:01.956885 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.956851 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6fv2w" event={"ID":"8756041d-afbd-4507-a82f-426bddd41acb","Type":"ContainerStarted","Data":"d8f653a39a55516f700325c7452cf5a6059d799f1b1476072a9be242f0b79f4c"} Apr 16 16:06:01.994007 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.993971 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-web-config\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994167 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994167 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994167 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994120 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994167 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994148 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994385 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994170 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994385 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994220 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994385 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994385 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrwd\" (UniqueName: \"kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-kube-api-access-nvrwd\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994385 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-config-volume\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994385 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994675 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:01.994675 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:01.994437 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-config-out\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.095702 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.095611 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-config-volume\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.095702 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.095664 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.095702 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.095689 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.095974 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.095707 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-config-out\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.095974 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.095737 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-web-config\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.095974 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.095755 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.095974 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.095877 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.095974 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.095921 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.095974 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.095958 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.096277 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.095985 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.096277 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.096043 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.096277 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.096083 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.096277 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.096120 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrwd\" (UniqueName: \"kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-kube-api-access-nvrwd\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.096480 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.096462 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.097246 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.097053 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.097246 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.097141 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.099249 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.099229 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.100053 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.099908 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.100053 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.099977 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-config-out\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.100053 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.099997 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.100265 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.100094 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-config-volume\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.100265 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.100131 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.100497 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.100476 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-web-config\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.101156 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.101138 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.101295 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.101277 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.108739 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.108716 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrwd\" (UniqueName: \"kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-kube-api-access-nvrwd\") pod \"alertmanager-main-0\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.200132 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.200094 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:06:02.350865 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.350782 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:06:02.354111 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:06:02.354083 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3effdb00_0466_4e49_97e5_e88860c7be16.slice/crio-3e8d0a991560bff8e0b07d122d296614c787bdbf0092c8d758e75754a59aafa1 WatchSource:0}: Error finding container 3e8d0a991560bff8e0b07d122d296614c787bdbf0092c8d758e75754a59aafa1: Status 404 returned error can't find the container with id 3e8d0a991560bff8e0b07d122d296614c787bdbf0092c8d758e75754a59aafa1 Apr 16 16:06:02.961643 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.961610 2576 generic.go:358] "Generic (PLEG): container finished" podID="8756041d-afbd-4507-a82f-426bddd41acb" containerID="58d74a8fac2a592e10fe901c3923789034e608af37976db8a525b544447205a7" exitCode=0 Apr 16 16:06:02.962061 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.961691 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6fv2w" event={"ID":"8756041d-afbd-4507-a82f-426bddd41acb","Type":"ContainerDied","Data":"58d74a8fac2a592e10fe901c3923789034e608af37976db8a525b544447205a7"} Apr 16 16:06:02.963348 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:02.963314 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerStarted","Data":"3e8d0a991560bff8e0b07d122d296614c787bdbf0092c8d758e75754a59aafa1"} Apr 16 16:06:03.878662 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.878569 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr"] Apr 16 16:06:03.891863 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.891835 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:03.894132 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.894104 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr"] Apr 16 16:06:03.894690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.894665 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-dqnn2cvdvavui\"" Apr 16 16:06:03.894832 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.894748 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-f4skq\"" Apr 16 16:06:03.894941 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.894898 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Apr 16 16:06:03.895252 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.895234 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Apr 16 16:06:03.897136 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.897115 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Apr 16 16:06:03.897136 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.897127 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Apr 16 16:06:03.897311 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.897115 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Apr 16 16:06:03.968063 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.968031 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6fv2w" event={"ID":"8756041d-afbd-4507-a82f-426bddd41acb","Type":"ContainerStarted","Data":"c138f905612f24bb215d1fc52a17db1bf0a05b7d8be6c54cdc307adaeb39afea"} Apr 16 16:06:03.968063 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.968069 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6fv2w" event={"ID":"8756041d-afbd-4507-a82f-426bddd41acb","Type":"ContainerStarted","Data":"f8f2365783ad98dde68db68415df54ad3a4fa59afdea22b1704a6607894b88d2"} Apr 16 16:06:03.969515 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.969483 2576 generic.go:358] "Generic (PLEG): container finished" podID="3effdb00-0466-4e49-97e5-e88860c7be16" containerID="8cc6572e7cf134cf55d343514925066344d155ddcdb8c17f63ab2142edc5d035" exitCode=0 Apr 16 16:06:03.969659 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.969529 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerDied","Data":"8cc6572e7cf134cf55d343514925066344d155ddcdb8c17f63ab2142edc5d035"} Apr 16 16:06:03.991126 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:03.991074 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6fv2w" podStartSLOduration=3.186952262 podStartE2EDuration="3.99105901s" podCreationTimestamp="2026-04-16 16:06:00 +0000 UTC" firstStartedPulling="2026-04-16 16:06:01.767533791 +0000 UTC m=+200.026280579" lastFinishedPulling="2026-04-16 16:06:02.571640537 +0000 UTC m=+200.830387327" observedRunningTime="2026-04-16 16:06:03.989836564 +0000 UTC m=+202.248583375" watchObservedRunningTime="2026-04-16 16:06:03.99105901 +0000 UTC m=+202.249805822" Apr 16 16:06:04.012831 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.012805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-grpc-tls\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.012947 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.012853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.013030 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.012978 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.013095 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.013032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.013244 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.013218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.013379 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.013358 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-tls\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.013461 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.013406 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc25446-af98-45ae-bf8c-d7747982bc50-metrics-client-ca\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.013709 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.013675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj78d\" (UniqueName: \"kubernetes.io/projected/3fc25446-af98-45ae-bf8c-d7747982bc50-kube-api-access-xj78d\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.114657 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.114620 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-tls\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.114810 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.114675 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc25446-af98-45ae-bf8c-d7747982bc50-metrics-client-ca\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.114810 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.114721 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xj78d\" (UniqueName: \"kubernetes.io/projected/3fc25446-af98-45ae-bf8c-d7747982bc50-kube-api-access-xj78d\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.114810 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.114752 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-grpc-tls\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.114810 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.114786 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.115028 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.114845 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.115028 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.114874 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.115028 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.114934 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.115423 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.115398 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fc25446-af98-45ae-bf8c-d7747982bc50-metrics-client-ca\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.117711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.117680 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.117784 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.117766 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.117920 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.117897 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-tls\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.117975 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.117956 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.118029 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.117987 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-grpc-tls\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.118442 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.118423 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/3fc25446-af98-45ae-bf8c-d7747982bc50-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.128572 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.128549 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj78d\" (UniqueName: \"kubernetes.io/projected/3fc25446-af98-45ae-bf8c-d7747982bc50-kube-api-access-xj78d\") pod \"thanos-querier-6cdb848f8b-zmtdr\" (UID: \"3fc25446-af98-45ae-bf8c-d7747982bc50\") " pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.203693 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.203613 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:04.327771 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.327724 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr"] Apr 16 16:06:04.331743 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:06:04.331719 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc25446_af98_45ae_bf8c_d7747982bc50.slice/crio-a88d96fdfc1bf90817e372e2b8b86362ef142e8f82ca2fdeb8987f25edbda8fb WatchSource:0}: Error finding container a88d96fdfc1bf90817e372e2b8b86362ef142e8f82ca2fdeb8987f25edbda8fb: Status 404 returned error can't find the container with id a88d96fdfc1bf90817e372e2b8b86362ef142e8f82ca2fdeb8987f25edbda8fb Apr 16 16:06:04.973201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:04.973162 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" event={"ID":"3fc25446-af98-45ae-bf8c-d7747982bc50","Type":"ContainerStarted","Data":"a88d96fdfc1bf90817e372e2b8b86362ef142e8f82ca2fdeb8987f25edbda8fb"} Apr 16 16:06:05.270198 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.270100 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6bbf66bb88-nl6hh"] Apr 16 16:06:05.274104 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.274007 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.278096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.278054 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Apr 16 16:06:05.278096 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.278090 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Apr 16 16:06:05.278308 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.278205 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Apr 16 16:06:05.278308 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.278226 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6s36bdd366k9e\"" Apr 16 16:06:05.278422 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.278348 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-s62nq\"" Apr 16 16:06:05.278422 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.278360 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Apr 16 16:06:05.290584 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.290560 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bbf66bb88-nl6hh"] Apr 16 16:06:05.325073 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.325040 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/60dee853-2b4a-44c9-a555-91259c774dbe-metrics-server-audit-profiles\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.325234 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.325096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60dee853-2b4a-44c9-a555-91259c774dbe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.325234 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.325162 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9spx\" (UniqueName: \"kubernetes.io/projected/60dee853-2b4a-44c9-a555-91259c774dbe-kube-api-access-v9spx\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.325357 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.325260 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/60dee853-2b4a-44c9-a555-91259c774dbe-secret-metrics-server-client-certs\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.325357 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.325303 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/60dee853-2b4a-44c9-a555-91259c774dbe-audit-log\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.325452 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.325412 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/60dee853-2b4a-44c9-a555-91259c774dbe-secret-metrics-server-tls\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.325502 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.325459 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/60dee853-2b4a-44c9-a555-91259c774dbe-client-ca-bundle\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.426408 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.426371 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/60dee853-2b4a-44c9-a555-91259c774dbe-secret-metrics-server-tls\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.426613 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.426427 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/60dee853-2b4a-44c9-a555-91259c774dbe-client-ca-bundle\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.426613 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.426473 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/60dee853-2b4a-44c9-a555-91259c774dbe-metrics-server-audit-profiles\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.426613 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.426525 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60dee853-2b4a-44c9-a555-91259c774dbe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.426613 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.426545 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v9spx\" (UniqueName: \"kubernetes.io/projected/60dee853-2b4a-44c9-a555-91259c774dbe-kube-api-access-v9spx\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.426613 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.426608 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/60dee853-2b4a-44c9-a555-91259c774dbe-secret-metrics-server-client-certs\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.426862 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.426643 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/60dee853-2b4a-44c9-a555-91259c774dbe-audit-log\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.427163 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.427112 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/60dee853-2b4a-44c9-a555-91259c774dbe-audit-log\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.427368 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.427348 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60dee853-2b4a-44c9-a555-91259c774dbe-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.429505 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.429446 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/60dee853-2b4a-44c9-a555-91259c774dbe-metrics-server-audit-profiles\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.429505 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.429493 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/60dee853-2b4a-44c9-a555-91259c774dbe-client-ca-bundle\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.429681 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.429592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/60dee853-2b4a-44c9-a555-91259c774dbe-secret-metrics-server-tls\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.429681 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.429640 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/60dee853-2b4a-44c9-a555-91259c774dbe-secret-metrics-server-client-certs\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.439321 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.439298 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9spx\" (UniqueName: \"kubernetes.io/projected/60dee853-2b4a-44c9-a555-91259c774dbe-kube-api-access-v9spx\") pod \"metrics-server-6bbf66bb88-nl6hh\" (UID: \"60dee853-2b4a-44c9-a555-91259c774dbe\") " pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.586706 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.586677 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:05.744095 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.744049 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bbf66bb88-nl6hh"] Apr 16 16:06:05.747795 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:06:05.747762 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60dee853_2b4a_44c9_a555_91259c774dbe.slice/crio-51a2270c157393f697de17637786558260db922b89546578741742b70004cae2 WatchSource:0}: Error finding container 51a2270c157393f697de17637786558260db922b89546578741742b70004cae2: Status 404 returned error can't find the container with id 51a2270c157393f697de17637786558260db922b89546578741742b70004cae2 Apr 16 16:06:05.796813 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.796791 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d7958c46-l2zxs"] Apr 16 16:06:05.797068 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:06:05.797051 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" podUID="7ed80516-2524-487c-8c72-d7ded9a82159" Apr 16 16:06:05.978089 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.978047 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" event={"ID":"60dee853-2b4a-44c9-a555-91259c774dbe","Type":"ContainerStarted","Data":"51a2270c157393f697de17637786558260db922b89546578741742b70004cae2"} Apr 16 16:06:05.981467 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.981436 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerStarted","Data":"23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0"} Apr 16 16:06:05.981467 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.981456 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:06:05.981467 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.981470 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerStarted","Data":"11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800"} Apr 16 16:06:05.981711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.981483 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerStarted","Data":"74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1"} Apr 16 16:06:05.981711 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.981495 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerStarted","Data":"8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2"} Apr 16 16:06:05.987129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:05.987106 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:06:06.032638 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.032610 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-installation-pull-secrets\") pod \"7ed80516-2524-487c-8c72-d7ded9a82159\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " Apr 16 16:06:06.032790 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.032661 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ed80516-2524-487c-8c72-d7ded9a82159-ca-trust-extracted\") pod \"7ed80516-2524-487c-8c72-d7ded9a82159\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " Apr 16 16:06:06.032790 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.032709 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-bound-sa-token\") pod \"7ed80516-2524-487c-8c72-d7ded9a82159\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " Apr 16 16:06:06.032790 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.032766 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-image-registry-private-configuration\") pod \"7ed80516-2524-487c-8c72-d7ded9a82159\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " Apr 16 16:06:06.032954 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.032802 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-trusted-ca\") pod \"7ed80516-2524-487c-8c72-d7ded9a82159\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " Apr 16 16:06:06.032954 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.032827 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qc5c8\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-kube-api-access-qc5c8\") pod \"7ed80516-2524-487c-8c72-d7ded9a82159\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " Apr 16 16:06:06.032954 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.032869 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-registry-certificates\") pod \"7ed80516-2524-487c-8c72-d7ded9a82159\" (UID: \"7ed80516-2524-487c-8c72-d7ded9a82159\") " Apr 16 16:06:06.033483 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.033301 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ed80516-2524-487c-8c72-d7ded9a82159-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7ed80516-2524-487c-8c72-d7ded9a82159" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 16:06:06.033483 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.033442 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7ed80516-2524-487c-8c72-d7ded9a82159" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:06:06.033671 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.033507 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7ed80516-2524-487c-8c72-d7ded9a82159" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:06:06.035624 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.035591 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7ed80516-2524-487c-8c72-d7ded9a82159" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:06:06.035758 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.035736 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7ed80516-2524-487c-8c72-d7ded9a82159" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:06:06.035821 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.035773 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-kube-api-access-qc5c8" (OuterVolumeSpecName: "kube-api-access-qc5c8") pod "7ed80516-2524-487c-8c72-d7ded9a82159" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159"). InnerVolumeSpecName "kube-api-access-qc5c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:06:06.035888 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.035855 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7ed80516-2524-487c-8c72-d7ded9a82159" (UID: "7ed80516-2524-487c-8c72-d7ded9a82159"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:06:06.107040 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.106949 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-566567f95c-vxvnr"] Apr 16 16:06:06.110832 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.110812 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.113628 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.113608 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Apr 16 16:06:06.113806 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.113784 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-85mpt\"" Apr 16 16:06:06.113916 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.113905 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Apr 16 16:06:06.114063 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.113997 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Apr 16 16:06:06.114289 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.114258 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Apr 16 16:06:06.114381 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.114288 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Apr 16 16:06:06.120459 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.120440 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Apr 16 16:06:06.126267 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.126242 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-566567f95c-vxvnr"] Apr 16 16:06:06.134575 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.134545 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-image-registry-private-configuration\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:06:06.134690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.134580 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-trusted-ca\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:06:06.134690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.134597 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc5c8\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-kube-api-access-qc5c8\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:06:06.134690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.134612 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ed80516-2524-487c-8c72-d7ded9a82159-registry-certificates\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:06:06.134690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.134627 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ed80516-2524-487c-8c72-d7ded9a82159-installation-pull-secrets\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:06:06.134690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.134641 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ed80516-2524-487c-8c72-d7ded9a82159-ca-trust-extracted\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:06:06.134690 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.134674 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-bound-sa-token\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:06:06.235935 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.235901 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2531bf-9760-40c2-9a24-03df27efff16-serving-certs-ca-bundle\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.236140 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.235973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2531bf-9760-40c2-9a24-03df27efff16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.236140 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.236038 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-telemeter-client-tls\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.236140 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.236061 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n7dl\" (UniqueName: \"kubernetes.io/projected/7b2531bf-9760-40c2-9a24-03df27efff16-kube-api-access-2n7dl\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.236140 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.236086 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b2531bf-9760-40c2-9a24-03df27efff16-metrics-client-ca\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.236140 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.236130 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.236363 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.236183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-secret-telemeter-client\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.236363 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.236207 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-federate-client-tls\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.337334 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.337302 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.337504 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.337357 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-secret-telemeter-client\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.337504 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.337386 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-federate-client-tls\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.337504 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.337430 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2531bf-9760-40c2-9a24-03df27efff16-serving-certs-ca-bundle\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.337504 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.337494 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2531bf-9760-40c2-9a24-03df27efff16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.337689 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.337552 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-telemeter-client-tls\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.337689 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.337583 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2n7dl\" (UniqueName: \"kubernetes.io/projected/7b2531bf-9760-40c2-9a24-03df27efff16-kube-api-access-2n7dl\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.337689 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.337623 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b2531bf-9760-40c2-9a24-03df27efff16-metrics-client-ca\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.338334 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.338259 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2531bf-9760-40c2-9a24-03df27efff16-serving-certs-ca-bundle\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.338461 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.338343 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b2531bf-9760-40c2-9a24-03df27efff16-metrics-client-ca\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.338461 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.338379 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b2531bf-9760-40c2-9a24-03df27efff16-telemeter-trusted-ca-bundle\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.339933 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.339910 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.340068 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.340051 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-federate-client-tls\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.340115 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.340087 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-telemeter-client-tls\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.340428 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.340405 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b2531bf-9760-40c2-9a24-03df27efff16-secret-telemeter-client\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.346761 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.346737 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n7dl\" (UniqueName: \"kubernetes.io/projected/7b2531bf-9760-40c2-9a24-03df27efff16-kube-api-access-2n7dl\") pod \"telemeter-client-566567f95c-vxvnr\" (UID: \"7b2531bf-9760-40c2-9a24-03df27efff16\") " pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.439851 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.439823 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" Apr 16 16:06:06.611898 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.611868 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-566567f95c-vxvnr"] Apr 16 16:06:06.613581 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:06:06.613517 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2531bf_9760_40c2_9a24_03df27efff16.slice/crio-9760c264ec8154143e8f2feaf3bde1f91e079d87b3d4961c71f5c8a91de2b43d WatchSource:0}: Error finding container 9760c264ec8154143e8f2feaf3bde1f91e079d87b3d4961c71f5c8a91de2b43d: Status 404 returned error can't find the container with id 9760c264ec8154143e8f2feaf3bde1f91e079d87b3d4961c71f5c8a91de2b43d Apr 16 16:06:06.986854 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.986818 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" event={"ID":"7b2531bf-9760-40c2-9a24-03df27efff16","Type":"ContainerStarted","Data":"9760c264ec8154143e8f2feaf3bde1f91e079d87b3d4961c71f5c8a91de2b43d"} Apr 16 16:06:06.990908 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.990850 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerStarted","Data":"fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442"} Apr 16 16:06:06.994214 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.993862 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-58d7958c46-l2zxs" Apr 16 16:06:06.994214 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.993936 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" event={"ID":"3fc25446-af98-45ae-bf8c-d7747982bc50","Type":"ContainerStarted","Data":"39a75bddb089227a676a68583761760121905021dc81ac8502294fd909eca543"} Apr 16 16:06:06.994214 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.993969 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" event={"ID":"3fc25446-af98-45ae-bf8c-d7747982bc50","Type":"ContainerStarted","Data":"2b9d385dc2ec6cb2e67f8da4acbe98cfdcf473db3276e0886a443743cd533eb8"} Apr 16 16:06:06.994214 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:06.993984 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" event={"ID":"3fc25446-af98-45ae-bf8c-d7747982bc50","Type":"ContainerStarted","Data":"dc8e3881beee31f74c56327d64a40a252a7b3679e9cbec38a0bd4a4d23dcff67"} Apr 16 16:06:07.037128 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.037076 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-58d7958c46-l2zxs"] Apr 16 16:06:07.044031 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.043985 2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-58d7958c46-l2zxs"] Apr 16 16:06:07.145741 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.145473 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:06:07.146789 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.146747 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ed80516-2524-487c-8c72-d7ded9a82159-registry-tls\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:06:07.150165 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.150142 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.152957 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.152935 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Apr 16 16:06:07.153214 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.153198 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Apr 16 16:06:07.154320 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.154194 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Apr 16 16:06:07.154540 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.154517 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Apr 16 16:06:07.154665 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.154650 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Apr 16 16:06:07.154807 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.154790 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Apr 16 16:06:07.154807 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.154801 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Apr 16 16:06:07.154946 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.154807 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Apr 16 16:06:07.154946 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.154846 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Apr 16 16:06:07.155168 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.155151 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-78tmmpncrljqa\"" Apr 16 16:06:07.155246 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.155226 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lkd9l\"" Apr 16 16:06:07.155575 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.155557 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Apr 16 16:06:07.160302 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.159894 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Apr 16 16:06:07.161306 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.161285 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Apr 16 16:06:07.166467 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.166447 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:06:07.247643 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.247557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.247643 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.247607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.247643 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.247636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.247926 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.247718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.247926 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.247744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm9dn\" (UniqueName: \"kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-kube-api-access-gm9dn\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.247926 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.247773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.247926 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.247836 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.247926 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.247876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.247926 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.247916 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.248259 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.247954 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.248259 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.248006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config-out\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.248259 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.248046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.248259 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.248077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.248259 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.248131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.248259 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.248182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.248259 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.248213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.248259 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.248258 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-web-config\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.248613 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.248283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349009 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.348972 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349009 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349030 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349254 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-web-config\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349301 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349347 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349380 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349516 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349588 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349620 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gm9dn\" (UniqueName: \"kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-kube-api-access-gm9dn\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349658 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349703 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349734 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349775 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349800 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.349804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349810 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.351653 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349885 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config-out\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.351653 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349923 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.351653 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349952 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.351653 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.349989 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.351653 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.350755 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.351653 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.350822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.351653 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.351396 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.352725 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.352399 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-web-config\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.353154 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.353055 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.354503 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.354207 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config-out\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.354503 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.354224 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.354882 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.354760 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.355254 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.355220 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.355346 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.355330 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.355664 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.355622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.356182 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.356139 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.356846 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.356804 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.357339 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.357283 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.358148 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.358121 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.359436 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.359415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.368391 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.368373 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm9dn\" (UniqueName: \"kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-kube-api-access-gm9dn\") pod \"prometheus-k8s-0\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.463759 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.463723 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:07.778486 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:07.778436 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:06:08.001621 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.001581 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerStarted","Data":"de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab"} Apr 16 16:06:08.004566 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.004533 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" event={"ID":"3fc25446-af98-45ae-bf8c-d7747982bc50","Type":"ContainerStarted","Data":"9290c72294a86a408aef98cce682fb75ae523f455fe3e9e1f4a10bb698801ded"} Apr 16 16:06:08.004701 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.004569 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" event={"ID":"3fc25446-af98-45ae-bf8c-d7747982bc50","Type":"ContainerStarted","Data":"6f45c9ffa3453dfc07bd213e0e5376ccf3fc421b0ee0b10c0b1deceb19cd46d2"} Apr 16 16:06:08.004701 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.004584 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" event={"ID":"3fc25446-af98-45ae-bf8c-d7747982bc50","Type":"ContainerStarted","Data":"9bddf7e6e5f0eeeb0ad44d5055114919dc434299f7291390cf53fb658fab228b"} Apr 16 16:06:08.004824 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.004701 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:08.006300 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.006046 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" event={"ID":"60dee853-2b4a-44c9-a555-91259c774dbe","Type":"ContainerStarted","Data":"2065db37cf659469827f79cc7b3cc6c20a3e202f43cd83cda1f7dda0de8bfb7c"} Apr 16 16:06:08.007599 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.007571 2576 generic.go:358] "Generic (PLEG): container finished" podID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerID="c68d4f8a7a8a6d5461d2172da5ecd939ffc5b7c93e428b987b1859044b59ea1f" exitCode=0 Apr 16 16:06:08.007698 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.007628 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerDied","Data":"c68d4f8a7a8a6d5461d2172da5ecd939ffc5b7c93e428b987b1859044b59ea1f"} Apr 16 16:06:08.007698 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.007648 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerStarted","Data":"0dc60e917ee15dbf287ac1c5d07c56c190d8dda4c8de56c15fc2ce9f802ddcd8"} Apr 16 16:06:08.032922 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.032869 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.765399934 podStartE2EDuration="7.032854263s" podCreationTimestamp="2026-04-16 16:06:01 +0000 UTC" firstStartedPulling="2026-04-16 16:06:02.356632649 +0000 UTC m=+200.615379439" lastFinishedPulling="2026-04-16 16:06:07.624086966 +0000 UTC m=+205.882833768" observedRunningTime="2026-04-16 16:06:08.031657806 +0000 UTC m=+206.290404641" watchObservedRunningTime="2026-04-16 16:06:08.032854263 +0000 UTC m=+206.291601074" Apr 16 16:06:08.089345 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.089288 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" podStartSLOduration=1.7963800220000001 podStartE2EDuration="5.089267227s" podCreationTimestamp="2026-04-16 16:06:03 +0000 UTC" firstStartedPulling="2026-04-16 16:06:04.333488881 +0000 UTC m=+202.592235669" lastFinishedPulling="2026-04-16 16:06:07.626376075 +0000 UTC m=+205.885122874" observedRunningTime="2026-04-16 16:06:08.088521277 +0000 UTC m=+206.347268106" watchObservedRunningTime="2026-04-16 16:06:08.089267227 +0000 UTC m=+206.348014039" Apr 16 16:06:08.113612 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.113138 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" podStartSLOduration=1.238850886 podStartE2EDuration="3.11312019s" podCreationTimestamp="2026-04-16 16:06:05 +0000 UTC" firstStartedPulling="2026-04-16 16:06:05.749983176 +0000 UTC m=+204.008729967" lastFinishedPulling="2026-04-16 16:06:07.624252482 +0000 UTC m=+205.882999271" observedRunningTime="2026-04-16 16:06:08.111388273 +0000 UTC m=+206.370135085" watchObservedRunningTime="2026-04-16 16:06:08.11312019 +0000 UTC m=+206.371867012" Apr 16 16:06:08.422762 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:08.422690 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ed80516-2524-487c-8c72-d7ded9a82159" path="/var/lib/kubelet/pods/7ed80516-2524-487c-8c72-d7ded9a82159/volumes" Apr 16 16:06:09.013494 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:09.013411 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" event={"ID":"7b2531bf-9760-40c2-9a24-03df27efff16","Type":"ContainerStarted","Data":"8c61c69d43c7052f0231c4f4e8e64941e223b7542fa3c27fd554eca5438364a0"} Apr 16 16:06:09.013494 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:09.013453 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" event={"ID":"7b2531bf-9760-40c2-9a24-03df27efff16","Type":"ContainerStarted","Data":"153086d717d24727b0d0f68e99bed67349e01e833fc55f6ece97c3d348770947"} Apr 16 16:06:09.013494 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:09.013468 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" event={"ID":"7b2531bf-9760-40c2-9a24-03df27efff16","Type":"ContainerStarted","Data":"b36ad5617b8af39754707409f4b792aef801684648f453ab92fb8837881628cd"} Apr 16 16:06:09.040194 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:09.040146 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-566567f95c-vxvnr" podStartSLOduration=1.222717379 podStartE2EDuration="3.040132241s" podCreationTimestamp="2026-04-16 16:06:06 +0000 UTC" firstStartedPulling="2026-04-16 16:06:06.615282083 +0000 UTC m=+204.874028879" lastFinishedPulling="2026-04-16 16:06:08.432696947 +0000 UTC m=+206.691443741" observedRunningTime="2026-04-16 16:06:09.038122713 +0000 UTC m=+207.296869522" watchObservedRunningTime="2026-04-16 16:06:09.040132241 +0000 UTC m=+207.298879050" Apr 16 16:06:11.031724 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:11.031695 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerStarted","Data":"974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8"} Apr 16 16:06:11.032081 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:11.031731 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerStarted","Data":"12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98"} Apr 16 16:06:11.032081 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:11.031745 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerStarted","Data":"c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0"} Apr 16 16:06:11.032081 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:11.031755 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerStarted","Data":"29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc"} Apr 16 16:06:12.037539 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:12.037507 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerStarted","Data":"0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90"} Apr 16 16:06:12.037539 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:12.037542 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerStarted","Data":"e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a"} Apr 16 16:06:12.073700 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:12.073634 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.344849647 podStartE2EDuration="5.073616997s" podCreationTimestamp="2026-04-16 16:06:07 +0000 UTC" firstStartedPulling="2026-04-16 16:06:08.008801164 +0000 UTC m=+206.267547955" lastFinishedPulling="2026-04-16 16:06:10.7375685 +0000 UTC m=+208.996315305" observedRunningTime="2026-04-16 16:06:12.071404902 +0000 UTC m=+210.330151717" watchObservedRunningTime="2026-04-16 16:06:12.073616997 +0000 UTC m=+210.332363798" Apr 16 16:06:12.464662 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:12.464633 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:06:14.019977 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:14.019949 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6cdb848f8b-zmtdr" Apr 16 16:06:14.414743 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:14.414662 2576 scope.go:117] "RemoveContainer" containerID="be5abf022a642338fab76b36d8765ba74f40da64a13a069ae256f9e9f9274912" Apr 16 16:06:15.049123 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:15.049097 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:06:15.049548 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:15.049153 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" event={"ID":"9541790d-c119-472d-ab6a-4915d4065240","Type":"ContainerStarted","Data":"380e15816a6e52915068d50b3546bcd2318739f45ad43bbf2dafe5d67c2593c9"} Apr 16 16:06:15.049548 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:15.049409 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:06:15.054097 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:15.054077 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" Apr 16 16:06:15.076460 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:15.076399 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-d87b8d5fc-qb2g6" podStartSLOduration=54.187429452 podStartE2EDuration="57.076384229s" podCreationTimestamp="2026-04-16 16:05:18 +0000 UTC" firstStartedPulling="2026-04-16 16:05:18.946369338 +0000 UTC m=+157.205116139" lastFinishedPulling="2026-04-16 16:05:21.835324114 +0000 UTC m=+160.094070916" observedRunningTime="2026-04-16 16:06:15.071243473 +0000 UTC m=+213.329990282" watchObservedRunningTime="2026-04-16 16:06:15.076384229 +0000 UTC m=+213.335131039" Apr 16 16:06:25.587382 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:25.587344 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:25.587382 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:25.587384 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:39.120111 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:39.120068 2576 generic.go:358] "Generic (PLEG): container finished" podID="057a2387-92a7-47ee-8fbd-ff2d0c13ee1a" containerID="cd0358402bc9a7927251595612054680bf395a04e531871ce659d93ec57ac4fa" exitCode=0 Apr 16 16:06:39.120649 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:39.120133 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" event={"ID":"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a","Type":"ContainerDied","Data":"cd0358402bc9a7927251595612054680bf395a04e531871ce659d93ec57ac4fa"} Apr 16 16:06:39.120649 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:39.120552 2576 scope.go:117] "RemoveContainer" containerID="cd0358402bc9a7927251595612054680bf395a04e531871ce659d93ec57ac4fa" Apr 16 16:06:40.124616 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:40.124584 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-756bb7d76f-htwnr" event={"ID":"057a2387-92a7-47ee-8fbd-ff2d0c13ee1a","Type":"ContainerStarted","Data":"53ab104778d5f0b528dd690a3ffa0ffe9a504a95c7637c627d2d048a4fbbcac5"} Apr 16 16:06:45.592883 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:45.592853 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:45.601751 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:45.601724 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6bbf66bb88-nl6hh" Apr 16 16:06:48.150077 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:48.150046 2576 generic.go:358] "Generic (PLEG): container finished" podID="dbf86cc5-0e80-47e9-b081-8510a5c57973" containerID="7bc12f2c8168f6f43583aa83edaff3e8925cbc1f1c7c1d326af80d16ac2b1416" exitCode=0 Apr 16 16:06:48.150564 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:48.150120 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" event={"ID":"dbf86cc5-0e80-47e9-b081-8510a5c57973","Type":"ContainerDied","Data":"7bc12f2c8168f6f43583aa83edaff3e8925cbc1f1c7c1d326af80d16ac2b1416"} Apr 16 16:06:48.150628 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:48.150601 2576 scope.go:117] "RemoveContainer" containerID="7bc12f2c8168f6f43583aa83edaff3e8925cbc1f1c7c1d326af80d16ac2b1416" Apr 16 16:06:49.154604 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:49.154568 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5785d4fcdd-q2snw" event={"ID":"dbf86cc5-0e80-47e9-b081-8510a5c57973","Type":"ContainerStarted","Data":"a9d5fc6a3220e6a2c2d7707b8b0602b476cf41948a4ecdfc64ff2e03821b0423"} Apr 16 16:06:54.300802 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:54.300756 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:06:54.303153 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:54.303127 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd5f8939-2604-4ca4-8d1e-05dc7f86fa41-metrics-certs\") pod \"network-metrics-daemon-fpztb\" (UID: \"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41\") " pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:06:54.518186 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:54.518159 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hwq6n\"" Apr 16 16:06:54.525611 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:54.525593 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fpztb" Apr 16 16:06:54.645465 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:54.645441 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fpztb"] Apr 16 16:06:54.647728 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:06:54.647696 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5f8939_2604_4ca4_8d1e_05dc7f86fa41.slice/crio-236f3d6ed0d4943b2ee504bfdb292aa12f457b523ef66b916b383818722e0828 WatchSource:0}: Error finding container 236f3d6ed0d4943b2ee504bfdb292aa12f457b523ef66b916b383818722e0828: Status 404 returned error can't find the container with id 236f3d6ed0d4943b2ee504bfdb292aa12f457b523ef66b916b383818722e0828 Apr 16 16:06:55.172529 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:55.172487 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fpztb" event={"ID":"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41","Type":"ContainerStarted","Data":"236f3d6ed0d4943b2ee504bfdb292aa12f457b523ef66b916b383818722e0828"} Apr 16 16:06:56.176680 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:56.176642 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fpztb" event={"ID":"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41","Type":"ContainerStarted","Data":"91eaf79e2488f6d83c6829d22ffca7bf6f0e81466501e7915432fa5e11312678"} Apr 16 16:06:56.176680 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:56.176677 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fpztb" event={"ID":"cd5f8939-2604-4ca4-8d1e-05dc7f86fa41","Type":"ContainerStarted","Data":"1165380d31d856ac5841e798b5dd5d5efc7b5058a5da160ab47a0f6141a4dd75"} Apr 16 16:06:56.195694 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:06:56.195647 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fpztb" podStartSLOduration=253.218688007 podStartE2EDuration="4m14.195631385s" podCreationTimestamp="2026-04-16 16:02:42 +0000 UTC" firstStartedPulling="2026-04-16 16:06:54.64961413 +0000 UTC m=+252.908360918" lastFinishedPulling="2026-04-16 16:06:55.626557504 +0000 UTC m=+253.885304296" observedRunningTime="2026-04-16 16:06:56.193138201 +0000 UTC m=+254.451885010" watchObservedRunningTime="2026-04-16 16:06:56.195631385 +0000 UTC m=+254.454378230" Apr 16 16:07:07.464995 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:07.464899 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:07.480369 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:07.480344 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:08.229088 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:08.229061 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:21.311370 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:21.311337 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:07:21.311842 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:21.311770 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="alertmanager" containerID="cri-o://8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2" gracePeriod=120 Apr 16 16:07:21.311842 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:21.311833 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy-metric" containerID="cri-o://fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442" gracePeriod=120 Apr 16 16:07:21.312081 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:21.311872 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy-web" containerID="cri-o://11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800" gracePeriod=120 Apr 16 16:07:21.312081 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:21.311928 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="prom-label-proxy" containerID="cri-o://de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab" gracePeriod=120 Apr 16 16:07:21.312081 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:21.311929 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy" containerID="cri-o://23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0" gracePeriod=120 Apr 16 16:07:21.312081 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:21.311936 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="config-reloader" containerID="cri-o://74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1" gracePeriod=120 Apr 16 16:07:21.797974 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:21.797919 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-kmvkl" podUID="f4bb9693-c1c3-4ecc-b82d-2ea9913caa85" Apr 16 16:07:21.797974 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:21.797919 2576 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-mt4rq" podUID="ad7697a0-8cda-49df-861c-5762c30a7794" Apr 16 16:07:22.260060 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260002 2576 generic.go:358] "Generic (PLEG): container finished" podID="3effdb00-0466-4e49-97e5-e88860c7be16" containerID="de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab" exitCode=0 Apr 16 16:07:22.260060 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260043 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerDied","Data":"de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab"} Apr 16 16:07:22.260264 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260079 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerDied","Data":"fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442"} Apr 16 16:07:22.260264 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260052 2576 generic.go:358] "Generic (PLEG): container finished" podID="3effdb00-0466-4e49-97e5-e88860c7be16" containerID="fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442" exitCode=0 Apr 16 16:07:22.260264 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260097 2576 generic.go:358] "Generic (PLEG): container finished" podID="3effdb00-0466-4e49-97e5-e88860c7be16" containerID="23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0" exitCode=0 Apr 16 16:07:22.260264 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260103 2576 generic.go:358] "Generic (PLEG): container finished" podID="3effdb00-0466-4e49-97e5-e88860c7be16" containerID="74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1" exitCode=0 Apr 16 16:07:22.260264 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260108 2576 generic.go:358] "Generic (PLEG): container finished" podID="3effdb00-0466-4e49-97e5-e88860c7be16" containerID="8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2" exitCode=0 Apr 16 16:07:22.260264 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260194 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerDied","Data":"23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0"} Apr 16 16:07:22.260264 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260211 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerDied","Data":"74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1"} Apr 16 16:07:22.260264 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260224 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerDied","Data":"8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2"} Apr 16 16:07:22.260264 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260245 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mt4rq" Apr 16 16:07:22.260264 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.260249 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:07:22.560131 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.560109 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:22.672833 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.672801 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvrwd\" (UniqueName: \"kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-kube-api-access-nvrwd\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.672833 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.672837 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-metrics-client-ca\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673090 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.672854 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-tls-assets\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673090 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.672887 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-config-volume\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673090 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.672905 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-web-config\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673090 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.672921 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-web\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673090 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.672942 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-config-out\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673090 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.672988 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-cluster-tls-config\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673090 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.673025 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-main-tls\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673451 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.673099 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-main-db\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673451 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.673143 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-trusted-ca-bundle\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673451 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.673169 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673451 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.673211 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy\") pod \"3effdb00-0466-4e49-97e5-e88860c7be16\" (UID: \"3effdb00-0466-4e49-97e5-e88860c7be16\") " Apr 16 16:07:22.673451 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.673257 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:07:22.673746 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.673531 2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-metrics-client-ca\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.674232 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.673809 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 16:07:22.674232 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.674192 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:07:22.676133 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.676095 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-kube-api-access-nvrwd" (OuterVolumeSpecName: "kube-api-access-nvrwd") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "kube-api-access-nvrwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:07:22.676346 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.676320 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:07:22.676430 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.676404 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-config-volume" (OuterVolumeSpecName: "config-volume") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:22.676607 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.676576 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:22.676719 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.676682 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:22.676816 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.676792 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-config-out" (OuterVolumeSpecName: "config-out") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 16:07:22.677399 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.677377 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:22.677892 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.677864 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:22.680848 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.680820 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:22.687254 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.687223 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-web-config" (OuterVolumeSpecName: "web-config") pod "3effdb00-0466-4e49-97e5-e88860c7be16" (UID: "3effdb00-0466-4e49-97e5-e88860c7be16"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:22.774427 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774394 2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-config-volume\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.774427 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774421 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-web-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.774427 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774431 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.774649 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774442 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-config-out\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.774649 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774452 2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-cluster-tls-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.774649 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774462 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-main-tls\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.774649 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774471 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-main-db\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.774649 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774480 2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3effdb00-0466-4e49-97e5-e88860c7be16-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.774649 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774490 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.774649 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774500 2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3effdb00-0466-4e49-97e5-e88860c7be16-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.774649 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774510 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvrwd\" (UniqueName: \"kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-kube-api-access-nvrwd\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:22.774649 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:22.774518 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3effdb00-0466-4e49-97e5-e88860c7be16-tls-assets\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:23.265820 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.265776 2576 generic.go:358] "Generic (PLEG): container finished" podID="3effdb00-0466-4e49-97e5-e88860c7be16" containerID="11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800" exitCode=0 Apr 16 16:07:23.266037 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.265853 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerDied","Data":"11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800"} Apr 16 16:07:23.266037 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.265874 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.266037 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.265903 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3effdb00-0466-4e49-97e5-e88860c7be16","Type":"ContainerDied","Data":"3e8d0a991560bff8e0b07d122d296614c787bdbf0092c8d758e75754a59aafa1"} Apr 16 16:07:23.266037 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.265925 2576 scope.go:117] "RemoveContainer" containerID="de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab" Apr 16 16:07:23.274491 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.274475 2576 scope.go:117] "RemoveContainer" containerID="fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442" Apr 16 16:07:23.281560 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.281543 2576 scope.go:117] "RemoveContainer" containerID="23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0" Apr 16 16:07:23.288672 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.288655 2576 scope.go:117] "RemoveContainer" containerID="11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800" Apr 16 16:07:23.292826 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.292740 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:07:23.300575 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.300552 2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:07:23.301896 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.301866 2576 scope.go:117] "RemoveContainer" containerID="74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1" Apr 16 16:07:23.308792 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.308776 2576 scope.go:117] "RemoveContainer" containerID="8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2" Apr 16 16:07:23.315515 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.315496 2576 scope.go:117] "RemoveContainer" containerID="8cc6572e7cf134cf55d343514925066344d155ddcdb8c17f63ab2142edc5d035" Apr 16 16:07:23.322176 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.322161 2576 scope.go:117] "RemoveContainer" containerID="de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab" Apr 16 16:07:23.322446 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:23.322425 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab\": container with ID starting with de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab not found: ID does not exist" containerID="de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab" Apr 16 16:07:23.322495 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.322457 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab"} err="failed to get container status \"de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab\": rpc error: code = NotFound desc = could not find container \"de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab\": container with ID starting with de013ce137f04ef54474dfaa89f52adb272137ba09cb294b27c0df1edf0d48ab not found: ID does not exist" Apr 16 16:07:23.322546 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.322497 2576 scope.go:117] "RemoveContainer" containerID="fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442" Apr 16 16:07:23.322772 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:23.322753 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442\": container with ID starting with fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442 not found: ID does not exist" containerID="fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442" Apr 16 16:07:23.322839 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.322782 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442"} err="failed to get container status \"fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442\": rpc error: code = NotFound desc = could not find container \"fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442\": container with ID starting with fbe2d2c01fc61576c25eb5f85f20f10e6f1857e4cc23362a381dc3fa99b7e442 not found: ID does not exist" Apr 16 16:07:23.322839 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.322806 2576 scope.go:117] "RemoveContainer" containerID="23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0" Apr 16 16:07:23.323043 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:23.323009 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0\": container with ID starting with 23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0 not found: ID does not exist" containerID="23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0" Apr 16 16:07:23.323088 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.323049 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0"} err="failed to get container status \"23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0\": rpc error: code = NotFound desc = could not find container \"23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0\": container with ID starting with 23548e2d026d79006cf75792b37a7ec872c86502fe6b0869f1cb2edb48ae47a0 not found: ID does not exist" Apr 16 16:07:23.323088 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.323063 2576 scope.go:117] "RemoveContainer" containerID="11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800" Apr 16 16:07:23.323326 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:23.323310 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800\": container with ID starting with 11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800 not found: ID does not exist" containerID="11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800" Apr 16 16:07:23.323376 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.323329 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800"} err="failed to get container status \"11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800\": rpc error: code = NotFound desc = could not find container \"11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800\": container with ID starting with 11ee9132183f78286bbfb4bb556d7ca8e946cae8bbd4dd50278c1a01f26f3800 not found: ID does not exist" Apr 16 16:07:23.323376 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.323342 2576 scope.go:117] "RemoveContainer" containerID="74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1" Apr 16 16:07:23.323544 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:23.323528 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1\": container with ID starting with 74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1 not found: ID does not exist" containerID="74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1" Apr 16 16:07:23.323585 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.323549 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1"} err="failed to get container status \"74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1\": rpc error: code = NotFound desc = could not find container \"74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1\": container with ID starting with 74431bbafd5f0becf9743a16f048a1f5f117766c48bc814ee7fc8e3043ca4fe1 not found: ID does not exist" Apr 16 16:07:23.323585 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.323562 2576 scope.go:117] "RemoveContainer" containerID="8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2" Apr 16 16:07:23.323758 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:23.323742 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2\": container with ID starting with 8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2 not found: ID does not exist" containerID="8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2" Apr 16 16:07:23.323816 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.323766 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2"} err="failed to get container status \"8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2\": rpc error: code = NotFound desc = could not find container \"8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2\": container with ID starting with 8d74f90d284de20fc88a50dc24af3d2be3d2f4520cac05b6df244df2b2622ab2 not found: ID does not exist" Apr 16 16:07:23.323816 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.323787 2576 scope.go:117] "RemoveContainer" containerID="8cc6572e7cf134cf55d343514925066344d155ddcdb8c17f63ab2142edc5d035" Apr 16 16:07:23.324023 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:23.323990 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc6572e7cf134cf55d343514925066344d155ddcdb8c17f63ab2142edc5d035\": container with ID starting with 8cc6572e7cf134cf55d343514925066344d155ddcdb8c17f63ab2142edc5d035 not found: ID does not exist" containerID="8cc6572e7cf134cf55d343514925066344d155ddcdb8c17f63ab2142edc5d035" Apr 16 16:07:23.324069 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.324030 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc6572e7cf134cf55d343514925066344d155ddcdb8c17f63ab2142edc5d035"} err="failed to get container status \"8cc6572e7cf134cf55d343514925066344d155ddcdb8c17f63ab2142edc5d035\": rpc error: code = NotFound desc = could not find container \"8cc6572e7cf134cf55d343514925066344d155ddcdb8c17f63ab2142edc5d035\": container with ID starting with 8cc6572e7cf134cf55d343514925066344d155ddcdb8c17f63ab2142edc5d035 not found: ID does not exist" Apr 16 16:07:23.333721 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.333700 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:07:23.334051 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334038 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="config-reloader" Apr 16 16:07:23.334104 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334053 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="config-reloader" Apr 16 16:07:23.334104 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334072 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy-metric" Apr 16 16:07:23.334104 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334077 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy-metric" Apr 16 16:07:23.334104 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334091 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="init-config-reloader" Apr 16 16:07:23.334104 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334097 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="init-config-reloader" Apr 16 16:07:23.334104 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334104 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="prom-label-proxy" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334109 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="prom-label-proxy" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334119 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334125 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334131 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy-web" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334136 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy-web" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334144 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="alertmanager" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334149 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="alertmanager" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334200 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy-metric" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334208 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="alertmanager" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334214 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334223 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="kube-rbac-proxy-web" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334232 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="config-reloader" Apr 16 16:07:23.334431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.334238 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" containerName="prom-label-proxy" Apr 16 16:07:23.339569 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.339554 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.342754 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.342737 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Apr 16 16:07:23.342831 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.342744 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Apr 16 16:07:23.343211 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.343182 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Apr 16 16:07:23.343300 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.343269 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Apr 16 16:07:23.343756 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.343736 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rzvwb\"" Apr 16 16:07:23.344063 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.344040 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Apr 16 16:07:23.344172 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.344052 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Apr 16 16:07:23.344238 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.344095 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Apr 16 16:07:23.344344 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.344320 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Apr 16 16:07:23.349866 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.349848 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Apr 16 16:07:23.354862 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.354842 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:07:23.379576 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25334c8e-c613-4d85-848f-f5c98c784a0c-config-out\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.379576 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379465 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.379576 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379495 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25334c8e-c613-4d85-848f-f5c98c784a0c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.379576 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379527 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.379576 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-web-config\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.380055 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.380055 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379688 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.380055 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379711 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr8zw\" (UniqueName: \"kubernetes.io/projected/25334c8e-c613-4d85-848f-f5c98c784a0c-kube-api-access-xr8zw\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.380055 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379752 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25334c8e-c613-4d85-848f-f5c98c784a0c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.380055 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379793 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.380055 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379822 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25334c8e-c613-4d85-848f-f5c98c784a0c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.380055 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-config-volume\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.380055 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.379871 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25334c8e-c613-4d85-848f-f5c98c784a0c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.480995 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.480957 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25334c8e-c613-4d85-848f-f5c98c784a0c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481192 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481005 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481192 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481156 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25334c8e-c613-4d85-848f-f5c98c784a0c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481265 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481206 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-config-volume\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481265 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481232 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25334c8e-c613-4d85-848f-f5c98c784a0c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481344 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481271 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25334c8e-c613-4d85-848f-f5c98c784a0c-config-out\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481344 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481326 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481464 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481429 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25334c8e-c613-4d85-848f-f5c98c784a0c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481520 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481495 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481579 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481532 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-web-config\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481639 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481624 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481689 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481658 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481689 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481684 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xr8zw\" (UniqueName: \"kubernetes.io/projected/25334c8e-c613-4d85-848f-f5c98c784a0c-kube-api-access-xr8zw\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.481954 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.481927 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25334c8e-c613-4d85-848f-f5c98c784a0c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.482155 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.482133 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25334c8e-c613-4d85-848f-f5c98c784a0c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.482960 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.482917 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/25334c8e-c613-4d85-848f-f5c98c784a0c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.484026 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.483955 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/25334c8e-c613-4d85-848f-f5c98c784a0c-config-out\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.484256 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.484233 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.484346 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.484261 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.484551 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.484524 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-web-config\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.484770 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.484749 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/25334c8e-c613-4d85-848f-f5c98c784a0c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.485075 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.485053 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.485496 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.485466 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.485721 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.485702 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-config-volume\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.485794 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.485776 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/25334c8e-c613-4d85-848f-f5c98c784a0c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.490089 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.490073 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr8zw\" (UniqueName: \"kubernetes.io/projected/25334c8e-c613-4d85-848f-f5c98c784a0c-kube-api-access-xr8zw\") pod \"alertmanager-main-0\" (UID: \"25334c8e-c613-4d85-848f-f5c98c784a0c\") " pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.649810 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.649711 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Apr 16 16:07:23.781912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:23.781887 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Apr 16 16:07:23.784299 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:07:23.784268 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25334c8e_c613_4d85_848f_f5c98c784a0c.slice/crio-180b826869bab0f6da25409aa3f090cd8fcf8d6362196afa95c271d635811d58 WatchSource:0}: Error finding container 180b826869bab0f6da25409aa3f090cd8fcf8d6362196afa95c271d635811d58: Status 404 returned error can't find the container with id 180b826869bab0f6da25409aa3f090cd8fcf8d6362196afa95c271d635811d58 Apr 16 16:07:24.271510 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:24.271475 2576 generic.go:358] "Generic (PLEG): container finished" podID="25334c8e-c613-4d85-848f-f5c98c784a0c" containerID="1e1fe817fd621325ab982633d574b1c851d93e2d70abe92f870a7b40df2638e8" exitCode=0 Apr 16 16:07:24.271510 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:24.271514 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25334c8e-c613-4d85-848f-f5c98c784a0c","Type":"ContainerDied","Data":"1e1fe817fd621325ab982633d574b1c851d93e2d70abe92f870a7b40df2638e8"} Apr 16 16:07:24.271863 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:24.271534 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25334c8e-c613-4d85-848f-f5c98c784a0c","Type":"ContainerStarted","Data":"180b826869bab0f6da25409aa3f090cd8fcf8d6362196afa95c271d635811d58"} Apr 16 16:07:24.419528 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:24.419499 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3effdb00-0466-4e49-97e5-e88860c7be16" path="/var/lib/kubelet/pods/3effdb00-0466-4e49-97e5-e88860c7be16/volumes" Apr 16 16:07:25.282169 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.282129 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25334c8e-c613-4d85-848f-f5c98c784a0c","Type":"ContainerStarted","Data":"dc56c1bce03fdbf7601d47a9db3d87ade4a197e9d1030e1e7ceb86a10b337b2e"} Apr 16 16:07:25.282169 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.282170 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25334c8e-c613-4d85-848f-f5c98c784a0c","Type":"ContainerStarted","Data":"7b3af8e75271ce2aefaa9a0836cb50532ebd1ef0b0e981103b3b960f5b490e73"} Apr 16 16:07:25.282585 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.282179 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25334c8e-c613-4d85-848f-f5c98c784a0c","Type":"ContainerStarted","Data":"5b39aa4af60fee02f23e189bda2e5f842992682dac338cff178a9a3f81768d86"} Apr 16 16:07:25.282585 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.282188 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25334c8e-c613-4d85-848f-f5c98c784a0c","Type":"ContainerStarted","Data":"f8f865c6ac96c9b3b79480814ff985217a70c6ca34d8736ff2d5aca5cad16273"} Apr 16 16:07:25.282585 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.282196 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25334c8e-c613-4d85-848f-f5c98c784a0c","Type":"ContainerStarted","Data":"b8f6a59826db8598cbe36bdf4c559aa5ba41e8fdb020d7dcf3e315934333916d"} Apr 16 16:07:25.282585 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.282204 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"25334c8e-c613-4d85-848f-f5c98c784a0c","Type":"ContainerStarted","Data":"507b919eeb7d52925e290992a95c8253b52550d00eb2360baa0daa752e08216b"} Apr 16 16:07:25.316003 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.315941 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.315925049 podStartE2EDuration="2.315925049s" podCreationTimestamp="2026-04-16 16:07:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:07:25.31456858 +0000 UTC m=+283.573315394" watchObservedRunningTime="2026-04-16 16:07:25.315925049 +0000 UTC m=+283.574671883" Apr 16 16:07:25.546914 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.546818 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:07:25.547458 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.547425 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="prometheus" containerID="cri-o://29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc" gracePeriod=600 Apr 16 16:07:25.547544 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.547462 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy-web" containerID="cri-o://974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8" gracePeriod=600 Apr 16 16:07:25.547544 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.547463 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="thanos-sidecar" containerID="cri-o://12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98" gracePeriod=600 Apr 16 16:07:25.547544 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.547483 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="config-reloader" containerID="cri-o://c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0" gracePeriod=600 Apr 16 16:07:25.547699 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.547483 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy" containerID="cri-o://e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a" gracePeriod=600 Apr 16 16:07:25.547811 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.547433 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy-thanos" containerID="cri-o://0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90" gracePeriod=600 Apr 16 16:07:25.603668 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.603646 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:07:25.603748 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.603735 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:07:25.606008 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.605986 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ad7697a0-8cda-49df-861c-5762c30a7794-metrics-tls\") pod \"dns-default-mt4rq\" (UID: \"ad7697a0-8cda-49df-861c-5762c30a7794\") " pod="openshift-dns/dns-default-mt4rq" Apr 16 16:07:25.606174 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.606155 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4bb9693-c1c3-4ecc-b82d-2ea9913caa85-cert\") pod \"ingress-canary-kmvkl\" (UID: \"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85\") " pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:07:25.865116 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.864999 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-l2qxk\"" Apr 16 16:07:25.865260 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.865126 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jhn24\"" Apr 16 16:07:25.871358 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.871335 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmvkl" Apr 16 16:07:25.871456 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:25.871425 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mt4rq" Apr 16 16:07:26.009879 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.009856 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mt4rq"] Apr 16 16:07:26.012342 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:07:26.012312 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad7697a0_8cda_49df_861c_5762c30a7794.slice/crio-208d383b2f8e94f2259793cb3cf9e9c860f18852a0875d45b86512004da5ebce WatchSource:0}: Error finding container 208d383b2f8e94f2259793cb3cf9e9c860f18852a0875d45b86512004da5ebce: Status 404 returned error can't find the container with id 208d383b2f8e94f2259793cb3cf9e9c860f18852a0875d45b86512004da5ebce Apr 16 16:07:26.023472 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.023450 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kmvkl"] Apr 16 16:07:26.025276 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:07:26.025253 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4bb9693_c1c3_4ecc_b82d_2ea9913caa85.slice/crio-45c8c9d8ee9bc1d9159073cecc02011dc0bd2a2faab585028b88458d3978b3b7 WatchSource:0}: Error finding container 45c8c9d8ee9bc1d9159073cecc02011dc0bd2a2faab585028b88458d3978b3b7: Status 404 returned error can't find the container with id 45c8c9d8ee9bc1d9159073cecc02011dc0bd2a2faab585028b88458d3978b3b7 Apr 16 16:07:26.288066 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.288027 2576 generic.go:358] "Generic (PLEG): container finished" podID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerID="0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90" exitCode=0 Apr 16 16:07:26.288066 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.288060 2576 generic.go:358] "Generic (PLEG): container finished" podID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerID="e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a" exitCode=0 Apr 16 16:07:26.288066 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.288062 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerDied","Data":"0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90"} Apr 16 16:07:26.288589 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.288093 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerDied","Data":"e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a"} Apr 16 16:07:26.288589 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.288104 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerDied","Data":"12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98"} Apr 16 16:07:26.288589 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.288071 2576 generic.go:358] "Generic (PLEG): container finished" podID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerID="12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98" exitCode=0 Apr 16 16:07:26.288589 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.288121 2576 generic.go:358] "Generic (PLEG): container finished" podID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerID="c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0" exitCode=0 Apr 16 16:07:26.288589 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.288127 2576 generic.go:358] "Generic (PLEG): container finished" podID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerID="29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc" exitCode=0 Apr 16 16:07:26.288589 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.288172 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerDied","Data":"c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0"} Apr 16 16:07:26.288589 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.288183 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerDied","Data":"29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc"} Apr 16 16:07:26.289238 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.289215 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mt4rq" event={"ID":"ad7697a0-8cda-49df-861c-5762c30a7794","Type":"ContainerStarted","Data":"208d383b2f8e94f2259793cb3cf9e9c860f18852a0875d45b86512004da5ebce"} Apr 16 16:07:26.290102 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.290080 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kmvkl" event={"ID":"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85","Type":"ContainerStarted","Data":"45c8c9d8ee9bc1d9159073cecc02011dc0bd2a2faab585028b88458d3978b3b7"} Apr 16 16:07:26.834028 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.833875 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:26.917436 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917347 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.917436 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917397 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm9dn\" (UniqueName: \"kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-kube-api-access-gm9dn\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.917436 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917424 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-grpc-tls\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.917736 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917458 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-db\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.917736 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917501 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-serving-certs-ca-bundle\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.917736 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917525 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-kube-rbac-proxy\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.917736 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917565 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-metrics-client-ca\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.917736 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917592 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.917736 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917665 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-thanos-prometheus-http-client-file\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.917736 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917699 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.918112 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917748 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config-out\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.918112 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917786 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-metrics-client-certs\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.918112 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917813 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-web-config\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.918112 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917845 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-tls-assets\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.918112 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917890 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-rulefiles-0\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.918112 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917948 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-trusted-ca-bundle\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.918112 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.917998 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-kubelet-serving-ca-bundle\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.918112 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.918051 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-tls\") pod \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\" (UID: \"11ae429c-0c38-4c83-a6f6-336d4dcf4554\") " Apr 16 16:07:26.920794 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.918669 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:07:26.920794 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.920009 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:07:26.920794 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.920398 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 16:07:26.922417 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.921226 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:07:26.922417 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.922134 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:07:26.922720 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.922682 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:07:26.924791 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.922812 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:26.924791 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.923221 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Apr 16 16:07:26.924791 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.924742 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:26.926232 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.926208 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config" (OuterVolumeSpecName: "config") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:26.926369 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.926348 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:26.926487 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.926466 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:26.926687 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.926651 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-kube-api-access-gm9dn" (OuterVolumeSpecName: "kube-api-access-gm9dn") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "kube-api-access-gm9dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:07:26.927367 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.927271 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:26.927367 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.927282 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:26.927367 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.927346 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:26.928143 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.928076 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config-out" (OuterVolumeSpecName: "config-out") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 16:07:26.946237 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:26.946204 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-web-config" (OuterVolumeSpecName: "web-config") pod "11ae429c-0c38-4c83-a6f6-336d4dcf4554" (UID: "11ae429c-0c38-4c83-a6f6-336d4dcf4554"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Apr 16 16:07:27.018845 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018808 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.018845 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018842 2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config-out\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.018845 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018854 2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-metrics-client-certs\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018863 2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-web-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018872 2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-tls-assets\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018880 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018890 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018900 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018911 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018923 2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018932 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm9dn\" (UniqueName: \"kubernetes.io/projected/11ae429c-0c38-4c83-a6f6-336d4dcf4554-kube-api-access-gm9dn\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018941 2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-grpc-tls\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018956 2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/11ae429c-0c38-4c83-a6f6-336d4dcf4554-prometheus-k8s-db\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018965 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018981 2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-secret-kube-rbac-proxy\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.018991 2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11ae429c-0c38-4c83-a6f6-336d4dcf4554-configmap-metrics-client-ca\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.019001 2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-config\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.019129 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.019025 2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/11ae429c-0c38-4c83-a6f6-336d4dcf4554-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:07:27.298305 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.298256 2576 generic.go:358] "Generic (PLEG): container finished" podID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerID="974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8" exitCode=0 Apr 16 16:07:27.298766 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.298341 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerDied","Data":"974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8"} Apr 16 16:07:27.298766 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.298393 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"11ae429c-0c38-4c83-a6f6-336d4dcf4554","Type":"ContainerDied","Data":"0dc60e917ee15dbf287ac1c5d07c56c190d8dda4c8de56c15fc2ce9f802ddcd8"} Apr 16 16:07:27.298766 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.298401 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.298766 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.298416 2576 scope.go:117] "RemoveContainer" containerID="0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90" Apr 16 16:07:27.327761 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.327717 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:07:27.332148 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.332114 2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:07:27.365493 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365459 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:07:27.365825 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365809 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="config-reloader" Apr 16 16:07:27.365880 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365829 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="config-reloader" Apr 16 16:07:27.365880 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365841 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy-web" Apr 16 16:07:27.365880 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365847 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy-web" Apr 16 16:07:27.365880 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365859 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="thanos-sidecar" Apr 16 16:07:27.365880 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365864 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="thanos-sidecar" Apr 16 16:07:27.365880 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365873 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy-thanos" Apr 16 16:07:27.365880 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365879 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy-thanos" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365896 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="prometheus" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365902 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="prometheus" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365909 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="init-config-reloader" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365914 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="init-config-reloader" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365921 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365927 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365985 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="config-reloader" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.365994 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy-web" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.366002 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.366029 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="thanos-sidecar" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.366038 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="kube-rbac-proxy-thanos" Apr 16 16:07:27.366120 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.366045 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" containerName="prometheus" Apr 16 16:07:27.370456 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.370431 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.373607 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.373587 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Apr 16 16:07:27.373878 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.373857 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lkd9l\"" Apr 16 16:07:27.373878 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.373866 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Apr 16 16:07:27.374109 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.373957 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Apr 16 16:07:27.374199 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.374173 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Apr 16 16:07:27.374340 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.374327 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Apr 16 16:07:27.375275 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.375257 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Apr 16 16:07:27.375384 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.375344 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Apr 16 16:07:27.375384 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.375376 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Apr 16 16:07:27.375554 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.375400 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Apr 16 16:07:27.375554 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.375452 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Apr 16 16:07:27.375948 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.375888 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-78tmmpncrljqa\"" Apr 16 16:07:27.378670 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.378652 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Apr 16 16:07:27.383741 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.382648 2576 scope.go:117] "RemoveContainer" containerID="e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a" Apr 16 16:07:27.385558 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.384097 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Apr 16 16:07:27.390804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.390773 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:07:27.423545 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.423545 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.423759 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.423759 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9a6123c0-9be7-4f39-be1c-f5b766e57efd-config-out\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.423759 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.423759 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9a6123c0-9be7-4f39-be1c-f5b766e57efd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.423912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423780 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.423912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-config\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.423912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9a6123c0-9be7-4f39-be1c-f5b766e57efd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.423912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.423912 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423878 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.424147 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423914 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.424147 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423975 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.424147 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.423994 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.424147 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.424029 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvln6\" (UniqueName: \"kubernetes.io/projected/9a6123c0-9be7-4f39-be1c-f5b766e57efd-kube-api-access-fvln6\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.424147 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.424046 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.424147 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.424068 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-web-config\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.424147 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.424110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.524702 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.524655 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.524897 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.524710 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-web-config\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.524897 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.524754 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.524897 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.524814 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.524897 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.524842 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.524897 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.524876 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.524910 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9a6123c0-9be7-4f39-be1c-f5b766e57efd-config-out\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.524940 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.524968 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9a6123c0-9be7-4f39-be1c-f5b766e57efd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.524993 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.525051 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-config\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.525079 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9a6123c0-9be7-4f39-be1c-f5b766e57efd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.525110 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.525128 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.525154 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525201 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.525193 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525724 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.525551 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.525724 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.525611 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.526433 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.526171 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.528380 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.528256 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.528380 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.528361 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.528380 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.528372 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9a6123c0-9be7-4f39-be1c-f5b766e57efd-config-out\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.528686 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.528658 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.529111 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.529049 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.530248 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.529468 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-web-config\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.530248 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.529520 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9a6123c0-9be7-4f39-be1c-f5b766e57efd-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.530248 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.529525 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.530248 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.529590 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fvln6\" (UniqueName: \"kubernetes.io/projected/9a6123c0-9be7-4f39-be1c-f5b766e57efd-kube-api-access-fvln6\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.530248 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.529896 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9a6123c0-9be7-4f39-be1c-f5b766e57efd-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.530248 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.530107 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.530248 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.530206 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.530673 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.530653 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.531874 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.531817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.532131 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.532109 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9a6123c0-9be7-4f39-be1c-f5b766e57efd-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.532313 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.532263 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9a6123c0-9be7-4f39-be1c-f5b766e57efd-config\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.541714 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.541685 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvln6\" (UniqueName: \"kubernetes.io/projected/9a6123c0-9be7-4f39-be1c-f5b766e57efd-kube-api-access-fvln6\") pod \"prometheus-k8s-0\" (UID: \"9a6123c0-9be7-4f39-be1c-f5b766e57efd\") " pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.681313 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.681266 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:27.882039 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.881813 2576 scope.go:117] "RemoveContainer" containerID="974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8" Apr 16 16:07:27.912383 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.912361 2576 scope.go:117] "RemoveContainer" containerID="12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98" Apr 16 16:07:27.922485 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.922453 2576 scope.go:117] "RemoveContainer" containerID="c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0" Apr 16 16:07:27.940880 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.940857 2576 scope.go:117] "RemoveContainer" containerID="29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc" Apr 16 16:07:27.948992 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.948969 2576 scope.go:117] "RemoveContainer" containerID="c68d4f8a7a8a6d5461d2172da5ecd939ffc5b7c93e428b987b1859044b59ea1f" Apr 16 16:07:27.964704 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.964678 2576 scope.go:117] "RemoveContainer" containerID="0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90" Apr 16 16:07:27.965227 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:27.965081 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90\": container with ID starting with 0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90 not found: ID does not exist" containerID="0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90" Apr 16 16:07:27.965227 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.965122 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90"} err="failed to get container status \"0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90\": rpc error: code = NotFound desc = could not find container \"0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90\": container with ID starting with 0e41330f73aca19f0a1864cf6855a4e99e96607178936323e70c260cc939cc90 not found: ID does not exist" Apr 16 16:07:27.965227 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.965150 2576 scope.go:117] "RemoveContainer" containerID="e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a" Apr 16 16:07:27.965613 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:27.965505 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a\": container with ID starting with e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a not found: ID does not exist" containerID="e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a" Apr 16 16:07:27.965613 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.965534 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a"} err="failed to get container status \"e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a\": rpc error: code = NotFound desc = could not find container \"e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a\": container with ID starting with e586002c42e6a8630dd197baf2290f84dc472ef4ca9fcae2082ce02b2fd67c7a not found: ID does not exist" Apr 16 16:07:27.965613 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.965555 2576 scope.go:117] "RemoveContainer" containerID="974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8" Apr 16 16:07:27.966382 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:27.966289 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8\": container with ID starting with 974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8 not found: ID does not exist" containerID="974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8" Apr 16 16:07:27.966382 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.966314 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8"} err="failed to get container status \"974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8\": rpc error: code = NotFound desc = could not find container \"974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8\": container with ID starting with 974f6a85ba381a19bfa327a913981cbfdb5af5cad0f28189f47ae35864da3cb8 not found: ID does not exist" Apr 16 16:07:27.966382 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.966329 2576 scope.go:117] "RemoveContainer" containerID="12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98" Apr 16 16:07:27.966715 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:27.966690 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98\": container with ID starting with 12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98 not found: ID does not exist" containerID="12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98" Apr 16 16:07:27.966783 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.966724 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98"} err="failed to get container status \"12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98\": rpc error: code = NotFound desc = could not find container \"12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98\": container with ID starting with 12d6551f58f5108965c84eb84dd519bd9707f3151ba1d0996091ef07f9a9ee98 not found: ID does not exist" Apr 16 16:07:27.966783 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.966745 2576 scope.go:117] "RemoveContainer" containerID="c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0" Apr 16 16:07:27.967007 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:27.966985 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0\": container with ID starting with c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0 not found: ID does not exist" containerID="c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0" Apr 16 16:07:27.967109 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.967037 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0"} err="failed to get container status \"c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0\": rpc error: code = NotFound desc = could not find container \"c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0\": container with ID starting with c6c0a282c123dead859d2b8ef28704e04a791641e9a745d261b588f2f07d81f0 not found: ID does not exist" Apr 16 16:07:27.967109 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.967058 2576 scope.go:117] "RemoveContainer" containerID="29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc" Apr 16 16:07:27.967397 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:27.967326 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc\": container with ID starting with 29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc not found: ID does not exist" containerID="29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc" Apr 16 16:07:27.967397 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.967355 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc"} err="failed to get container status \"29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc\": rpc error: code = NotFound desc = could not find container \"29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc\": container with ID starting with 29677b31cc54b7ed4c05f648c73d3db54e6add0f377398aedca100ff82c753cc not found: ID does not exist" Apr 16 16:07:27.967397 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.967374 2576 scope.go:117] "RemoveContainer" containerID="c68d4f8a7a8a6d5461d2172da5ecd939ffc5b7c93e428b987b1859044b59ea1f" Apr 16 16:07:27.968070 ip-10-0-143-177 kubenswrapper[2576]: E0416 16:07:27.967974 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68d4f8a7a8a6d5461d2172da5ecd939ffc5b7c93e428b987b1859044b59ea1f\": container with ID starting with c68d4f8a7a8a6d5461d2172da5ecd939ffc5b7c93e428b987b1859044b59ea1f not found: ID does not exist" containerID="c68d4f8a7a8a6d5461d2172da5ecd939ffc5b7c93e428b987b1859044b59ea1f" Apr 16 16:07:27.968070 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:27.968055 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68d4f8a7a8a6d5461d2172da5ecd939ffc5b7c93e428b987b1859044b59ea1f"} err="failed to get container status \"c68d4f8a7a8a6d5461d2172da5ecd939ffc5b7c93e428b987b1859044b59ea1f\": rpc error: code = NotFound desc = could not find container \"c68d4f8a7a8a6d5461d2172da5ecd939ffc5b7c93e428b987b1859044b59ea1f\": container with ID starting with c68d4f8a7a8a6d5461d2172da5ecd939ffc5b7c93e428b987b1859044b59ea1f not found: ID does not exist" Apr 16 16:07:28.076950 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:28.076897 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Apr 16 16:07:28.078396 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:07:28.078371 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6123c0_9be7_4f39_be1c_f5b766e57efd.slice/crio-34ddec21f5513fa6dc52da0bfd842f4371a43b4fa1fa5889a714735cbe7b993f WatchSource:0}: Error finding container 34ddec21f5513fa6dc52da0bfd842f4371a43b4fa1fa5889a714735cbe7b993f: Status 404 returned error can't find the container with id 34ddec21f5513fa6dc52da0bfd842f4371a43b4fa1fa5889a714735cbe7b993f Apr 16 16:07:28.303545 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:28.303508 2576 generic.go:358] "Generic (PLEG): container finished" podID="9a6123c0-9be7-4f39-be1c-f5b766e57efd" containerID="0815c95c308ea41dbe44a1e943e5c9b1ac6419167878003ef9ebbe5d5c06cc5c" exitCode=0 Apr 16 16:07:28.304008 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:28.303598 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a6123c0-9be7-4f39-be1c-f5b766e57efd","Type":"ContainerDied","Data":"0815c95c308ea41dbe44a1e943e5c9b1ac6419167878003ef9ebbe5d5c06cc5c"} Apr 16 16:07:28.304008 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:28.303642 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a6123c0-9be7-4f39-be1c-f5b766e57efd","Type":"ContainerStarted","Data":"34ddec21f5513fa6dc52da0bfd842f4371a43b4fa1fa5889a714735cbe7b993f"} Apr 16 16:07:28.304984 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:28.304959 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kmvkl" event={"ID":"f4bb9693-c1c3-4ecc-b82d-2ea9913caa85","Type":"ContainerStarted","Data":"0febaf0027eeb0b80501e0af8abe445fdf21fdff2c4c456fef7ea5a6d6596648"} Apr 16 16:07:28.307440 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:28.307384 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mt4rq" event={"ID":"ad7697a0-8cda-49df-861c-5762c30a7794","Type":"ContainerStarted","Data":"1296b19e6ad9db95220675e4720b8c6d3add25f1ffb3dc1a28b254c580ab2d6e"} Apr 16 16:07:28.307440 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:28.307412 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mt4rq" event={"ID":"ad7697a0-8cda-49df-861c-5762c30a7794","Type":"ContainerStarted","Data":"d236b27b0bbe615a3128fe46f36a8b378a1ec729575bb2da29b6711d5aea0813"} Apr 16 16:07:28.307626 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:28.307523 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mt4rq" Apr 16 16:07:28.347429 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:28.347247 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kmvkl" podStartSLOduration=251.4452945 podStartE2EDuration="4m13.3472202s" podCreationTimestamp="2026-04-16 16:03:15 +0000 UTC" firstStartedPulling="2026-04-16 16:07:26.026844205 +0000 UTC m=+284.285590996" lastFinishedPulling="2026-04-16 16:07:27.928769891 +0000 UTC m=+286.187516696" observedRunningTime="2026-04-16 16:07:28.347059955 +0000 UTC m=+286.605806771" watchObservedRunningTime="2026-04-16 16:07:28.3472202 +0000 UTC m=+286.605967015" Apr 16 16:07:28.371137 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:28.371078 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-mt4rq" podStartSLOduration=251.458571245 podStartE2EDuration="4m13.371058028s" podCreationTimestamp="2026-04-16 16:03:15 +0000 UTC" firstStartedPulling="2026-04-16 16:07:26.014047832 +0000 UTC m=+284.272794623" lastFinishedPulling="2026-04-16 16:07:27.926534602 +0000 UTC m=+286.185281406" observedRunningTime="2026-04-16 16:07:28.369152174 +0000 UTC m=+286.627898989" watchObservedRunningTime="2026-04-16 16:07:28.371058028 +0000 UTC m=+286.629804838" Apr 16 16:07:28.419409 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:28.418714 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11ae429c-0c38-4c83-a6f6-336d4dcf4554" path="/var/lib/kubelet/pods/11ae429c-0c38-4c83-a6f6-336d4dcf4554/volumes" Apr 16 16:07:29.314581 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:29.314545 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a6123c0-9be7-4f39-be1c-f5b766e57efd","Type":"ContainerStarted","Data":"b3c432585e35e9f29b14ebd5d287927b5483c15286a58cd2af094b956f3c4266"} Apr 16 16:07:29.314581 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:29.314585 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a6123c0-9be7-4f39-be1c-f5b766e57efd","Type":"ContainerStarted","Data":"4b8bf2235390eb10639753c949937b01a7384f47d6f96de1d391003a320bbe91"} Apr 16 16:07:29.314581 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:29.314594 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a6123c0-9be7-4f39-be1c-f5b766e57efd","Type":"ContainerStarted","Data":"4ca4ae915537c8a5da628bea5ecbf02250cad9d09e9a6b8d997f8a0fd25d7493"} Apr 16 16:07:29.314581 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:29.314603 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a6123c0-9be7-4f39-be1c-f5b766e57efd","Type":"ContainerStarted","Data":"22e9acd692dd0a67b49a35a84455acb67f8133f3d2088f3d4a96125b298d929f"} Apr 16 16:07:29.315160 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:29.314612 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a6123c0-9be7-4f39-be1c-f5b766e57efd","Type":"ContainerStarted","Data":"9943b8b808a9aac9de45f35aed17302ca44d677ed4d6ca72ab41a9ae1fceeb88"} Apr 16 16:07:29.315160 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:29.314620 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9a6123c0-9be7-4f39-be1c-f5b766e57efd","Type":"ContainerStarted","Data":"988bb22d6e206fa6ed18d2602bf218995d7cfc7f6fe7b15998846aec15af28e9"} Apr 16 16:07:29.349053 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:29.348976 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.3489538469999998 podStartE2EDuration="2.348953847s" podCreationTimestamp="2026-04-16 16:07:27 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:07:29.346574292 +0000 UTC m=+287.605321115" watchObservedRunningTime="2026-04-16 16:07:29.348953847 +0000 UTC m=+287.607700661" Apr 16 16:07:32.681669 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:32.681628 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:07:38.317279 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:38.317248 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mt4rq" Apr 16 16:07:42.311180 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:42.311155 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:07:42.311598 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:42.311581 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:07:42.321989 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:42.321966 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:07:42.322232 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:07:42.322216 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:08:27.681597 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:08:27.681564 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:08:27.696706 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:08:27.696684 2576 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:08:28.503673 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:08:28.503645 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Apr 16 16:12:16.471949 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.471913 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-init-hl4hx"] Apr 16 16:12:16.475349 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.475332 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hl4hx" Apr 16 16:12:16.477950 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.477923 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Apr 16 16:12:16.477950 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.477942 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Apr 16 16:12:16.478218 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.477923 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Apr 16 16:12:16.478960 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.478923 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jdvqm\"" Apr 16 16:12:16.481441 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.481411 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hl4hx"] Apr 16 16:12:16.527373 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.527335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9fc4\" (UniqueName: \"kubernetes.io/projected/1527e1d7-a803-43d1-bcee-05d26181772f-kube-api-access-v9fc4\") pod \"s3-init-hl4hx\" (UID: \"1527e1d7-a803-43d1-bcee-05d26181772f\") " pod="kserve/s3-init-hl4hx" Apr 16 16:12:16.628265 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.628234 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-v9fc4\" (UniqueName: \"kubernetes.io/projected/1527e1d7-a803-43d1-bcee-05d26181772f-kube-api-access-v9fc4\") pod \"s3-init-hl4hx\" (UID: \"1527e1d7-a803-43d1-bcee-05d26181772f\") " pod="kserve/s3-init-hl4hx" Apr 16 16:12:16.636653 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.636622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9fc4\" (UniqueName: \"kubernetes.io/projected/1527e1d7-a803-43d1-bcee-05d26181772f-kube-api-access-v9fc4\") pod \"s3-init-hl4hx\" (UID: \"1527e1d7-a803-43d1-bcee-05d26181772f\") " pod="kserve/s3-init-hl4hx" Apr 16 16:12:16.795055 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.794946 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hl4hx" Apr 16 16:12:16.916362 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.916337 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hl4hx"] Apr 16 16:12:16.918885 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:12:16.918849 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1527e1d7_a803_43d1_bcee_05d26181772f.slice/crio-d6b5250488079320e7cfe9720309696789e579e40cfce42cabe9d2a76e938296 WatchSource:0}: Error finding container d6b5250488079320e7cfe9720309696789e579e40cfce42cabe9d2a76e938296: Status 404 returned error can't find the container with id d6b5250488079320e7cfe9720309696789e579e40cfce42cabe9d2a76e938296 Apr 16 16:12:16.920669 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:16.920653 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 16:12:17.186112 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:17.186023 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hl4hx" event={"ID":"1527e1d7-a803-43d1-bcee-05d26181772f","Type":"ContainerStarted","Data":"d6b5250488079320e7cfe9720309696789e579e40cfce42cabe9d2a76e938296"} Apr 16 16:12:22.205035 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:22.204972 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hl4hx" event={"ID":"1527e1d7-a803-43d1-bcee-05d26181772f","Type":"ContainerStarted","Data":"994f452521f289627ce9b648087c0144fdc48f978f7734a983c2c23b8e2f1a1d"} Apr 16 16:12:22.220003 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:22.219948 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-init-hl4hx" podStartSLOduration=1.7249993959999999 podStartE2EDuration="6.219931533s" podCreationTimestamp="2026-04-16 16:12:16 +0000 UTC" firstStartedPulling="2026-04-16 16:12:16.920780749 +0000 UTC m=+575.179527540" lastFinishedPulling="2026-04-16 16:12:21.415712884 +0000 UTC m=+579.674459677" observedRunningTime="2026-04-16 16:12:22.21904785 +0000 UTC m=+580.477794858" watchObservedRunningTime="2026-04-16 16:12:22.219931533 +0000 UTC m=+580.478678343" Apr 16 16:12:25.214678 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:25.214642 2576 generic.go:358] "Generic (PLEG): container finished" podID="1527e1d7-a803-43d1-bcee-05d26181772f" containerID="994f452521f289627ce9b648087c0144fdc48f978f7734a983c2c23b8e2f1a1d" exitCode=0 Apr 16 16:12:25.215168 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:25.214708 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hl4hx" event={"ID":"1527e1d7-a803-43d1-bcee-05d26181772f","Type":"ContainerDied","Data":"994f452521f289627ce9b648087c0144fdc48f978f7734a983c2c23b8e2f1a1d"} Apr 16 16:12:26.351984 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:26.351961 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hl4hx" Apr 16 16:12:26.421605 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:26.421578 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9fc4\" (UniqueName: \"kubernetes.io/projected/1527e1d7-a803-43d1-bcee-05d26181772f-kube-api-access-v9fc4\") pod \"1527e1d7-a803-43d1-bcee-05d26181772f\" (UID: \"1527e1d7-a803-43d1-bcee-05d26181772f\") " Apr 16 16:12:26.423643 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:26.423617 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1527e1d7-a803-43d1-bcee-05d26181772f-kube-api-access-v9fc4" (OuterVolumeSpecName: "kube-api-access-v9fc4") pod "1527e1d7-a803-43d1-bcee-05d26181772f" (UID: "1527e1d7-a803-43d1-bcee-05d26181772f"). InnerVolumeSpecName "kube-api-access-v9fc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:12:26.523172 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:26.523053 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9fc4\" (UniqueName: \"kubernetes.io/projected/1527e1d7-a803-43d1-bcee-05d26181772f-kube-api-access-v9fc4\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:12:27.223042 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:27.222975 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hl4hx" event={"ID":"1527e1d7-a803-43d1-bcee-05d26181772f","Type":"ContainerDied","Data":"d6b5250488079320e7cfe9720309696789e579e40cfce42cabe9d2a76e938296"} Apr 16 16:12:27.223234 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:27.223062 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6b5250488079320e7cfe9720309696789e579e40cfce42cabe9d2a76e938296" Apr 16 16:12:27.223234 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:27.223164 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hl4hx" Apr 16 16:12:35.008214 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.008177 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-vtd2d"] Apr 16 16:12:35.008669 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.008521 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1527e1d7-a803-43d1-bcee-05d26181772f" containerName="s3-init" Apr 16 16:12:35.008669 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.008533 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1527e1d7-a803-43d1-bcee-05d26181772f" containerName="s3-init" Apr 16 16:12:35.008669 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.008594 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="1527e1d7-a803-43d1-bcee-05d26181772f" containerName="s3-init" Apr 16 16:12:35.010431 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.010415 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vtd2d" Apr 16 16:12:35.013039 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.012994 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\"" Apr 16 16:12:35.013039 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.013008 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Apr 16 16:12:35.013245 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.013091 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Apr 16 16:12:35.013245 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.013106 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jdvqm\"" Apr 16 16:12:35.018595 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.018576 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-vtd2d"] Apr 16 16:12:35.095115 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.095078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kktq7\" (UniqueName: \"kubernetes.io/projected/85284f39-75fd-460f-85f4-5d2798acc482-kube-api-access-kktq7\") pod \"s3-tls-init-custom-vtd2d\" (UID: \"85284f39-75fd-460f-85f4-5d2798acc482\") " pod="kserve/s3-tls-init-custom-vtd2d" Apr 16 16:12:35.196040 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.195977 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kktq7\" (UniqueName: \"kubernetes.io/projected/85284f39-75fd-460f-85f4-5d2798acc482-kube-api-access-kktq7\") pod \"s3-tls-init-custom-vtd2d\" (UID: \"85284f39-75fd-460f-85f4-5d2798acc482\") " pod="kserve/s3-tls-init-custom-vtd2d" Apr 16 16:12:35.203821 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.203790 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kktq7\" (UniqueName: \"kubernetes.io/projected/85284f39-75fd-460f-85f4-5d2798acc482-kube-api-access-kktq7\") pod \"s3-tls-init-custom-vtd2d\" (UID: \"85284f39-75fd-460f-85f4-5d2798acc482\") " pod="kserve/s3-tls-init-custom-vtd2d" Apr 16 16:12:35.331514 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.331429 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vtd2d" Apr 16 16:12:35.450531 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:35.450505 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-vtd2d"] Apr 16 16:12:35.452621 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:12:35.452582 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85284f39_75fd_460f_85f4_5d2798acc482.slice/crio-c3c78c5853ffa90b40af50d5f528ddd89a3f6cfd58a441b7c37f96d1c669d50a WatchSource:0}: Error finding container c3c78c5853ffa90b40af50d5f528ddd89a3f6cfd58a441b7c37f96d1c669d50a: Status 404 returned error can't find the container with id c3c78c5853ffa90b40af50d5f528ddd89a3f6cfd58a441b7c37f96d1c669d50a Apr 16 16:12:36.251734 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:36.251697 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vtd2d" event={"ID":"85284f39-75fd-460f-85f4-5d2798acc482","Type":"ContainerStarted","Data":"2fb560a3bea117303567541b80e18bbf3c82415edf38b67398d38a86df044847"} Apr 16 16:12:36.251734 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:36.251734 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vtd2d" event={"ID":"85284f39-75fd-460f-85f4-5d2798acc482","Type":"ContainerStarted","Data":"c3c78c5853ffa90b40af50d5f528ddd89a3f6cfd58a441b7c37f96d1c669d50a"} Apr 16 16:12:36.266894 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:36.266851 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-vtd2d" podStartSLOduration=2.266836 podStartE2EDuration="2.266836s" podCreationTimestamp="2026-04-16 16:12:34 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:12:36.265957713 +0000 UTC m=+594.524704524" watchObservedRunningTime="2026-04-16 16:12:36.266836 +0000 UTC m=+594.525582809" Apr 16 16:12:41.267449 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:41.267411 2576 generic.go:358] "Generic (PLEG): container finished" podID="85284f39-75fd-460f-85f4-5d2798acc482" containerID="2fb560a3bea117303567541b80e18bbf3c82415edf38b67398d38a86df044847" exitCode=0 Apr 16 16:12:41.267900 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:41.267481 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vtd2d" event={"ID":"85284f39-75fd-460f-85f4-5d2798acc482","Type":"ContainerDied","Data":"2fb560a3bea117303567541b80e18bbf3c82415edf38b67398d38a86df044847"} Apr 16 16:12:42.350866 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:42.350840 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:12:42.352115 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:42.352089 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:12:42.356044 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:42.356007 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:12:42.357262 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:42.357242 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:12:42.399904 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:42.399885 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vtd2d" Apr 16 16:12:42.562117 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:42.562038 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kktq7\" (UniqueName: \"kubernetes.io/projected/85284f39-75fd-460f-85f4-5d2798acc482-kube-api-access-kktq7\") pod \"85284f39-75fd-460f-85f4-5d2798acc482\" (UID: \"85284f39-75fd-460f-85f4-5d2798acc482\") " Apr 16 16:12:42.563986 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:42.563963 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85284f39-75fd-460f-85f4-5d2798acc482-kube-api-access-kktq7" (OuterVolumeSpecName: "kube-api-access-kktq7") pod "85284f39-75fd-460f-85f4-5d2798acc482" (UID: "85284f39-75fd-460f-85f4-5d2798acc482"). InnerVolumeSpecName "kube-api-access-kktq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:12:42.663382 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:42.663348 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kktq7\" (UniqueName: \"kubernetes.io/projected/85284f39-75fd-460f-85f4-5d2798acc482-kube-api-access-kktq7\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:12:43.274648 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:43.274616 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-vtd2d" event={"ID":"85284f39-75fd-460f-85f4-5d2798acc482","Type":"ContainerDied","Data":"c3c78c5853ffa90b40af50d5f528ddd89a3f6cfd58a441b7c37f96d1c669d50a"} Apr 16 16:12:43.274648 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:43.274647 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c78c5853ffa90b40af50d5f528ddd89a3f6cfd58a441b7c37f96d1c669d50a" Apr 16 16:12:43.274648 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:43.274645 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-vtd2d" Apr 16 16:12:48.537432 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.537403 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-hhl44"] Apr 16 16:12:48.537805 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.537748 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85284f39-75fd-460f-85f4-5d2798acc482" containerName="s3-tls-init-custom" Apr 16 16:12:48.537805 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.537761 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="85284f39-75fd-460f-85f4-5d2798acc482" containerName="s3-tls-init-custom" Apr 16 16:12:48.537874 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.537833 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="85284f39-75fd-460f-85f4-5d2798acc482" containerName="s3-tls-init-custom" Apr 16 16:12:48.539748 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.539731 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hhl44" Apr 16 16:12:48.542368 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.542342 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Apr 16 16:12:48.542368 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.542342 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Apr 16 16:12:48.543380 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.543364 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\"" Apr 16 16:12:48.543438 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.543370 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-jdvqm\"" Apr 16 16:12:48.547035 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.547001 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-hhl44"] Apr 16 16:12:48.718438 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.718405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlg8j\" (UniqueName: \"kubernetes.io/projected/352f238d-e5f0-4530-8b2e-63f15ab29644-kube-api-access-zlg8j\") pod \"s3-tls-init-serving-hhl44\" (UID: \"352f238d-e5f0-4530-8b2e-63f15ab29644\") " pod="kserve/s3-tls-init-serving-hhl44" Apr 16 16:12:48.819216 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.819146 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zlg8j\" (UniqueName: \"kubernetes.io/projected/352f238d-e5f0-4530-8b2e-63f15ab29644-kube-api-access-zlg8j\") pod \"s3-tls-init-serving-hhl44\" (UID: \"352f238d-e5f0-4530-8b2e-63f15ab29644\") " pod="kserve/s3-tls-init-serving-hhl44" Apr 16 16:12:48.826911 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.826890 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlg8j\" (UniqueName: \"kubernetes.io/projected/352f238d-e5f0-4530-8b2e-63f15ab29644-kube-api-access-zlg8j\") pod \"s3-tls-init-serving-hhl44\" (UID: \"352f238d-e5f0-4530-8b2e-63f15ab29644\") " pod="kserve/s3-tls-init-serving-hhl44" Apr 16 16:12:48.861955 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.861920 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hhl44" Apr 16 16:12:48.975614 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:48.975582 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-hhl44"] Apr 16 16:12:48.978307 ip-10-0-143-177 kubenswrapper[2576]: W0416 16:12:48.978279 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod352f238d_e5f0_4530_8b2e_63f15ab29644.slice/crio-8719c24dab03b86abc8bfc279c366a4fbc7e2e32e79e104e37c97a318760f5b0 WatchSource:0}: Error finding container 8719c24dab03b86abc8bfc279c366a4fbc7e2e32e79e104e37c97a318760f5b0: Status 404 returned error can't find the container with id 8719c24dab03b86abc8bfc279c366a4fbc7e2e32e79e104e37c97a318760f5b0 Apr 16 16:12:49.296524 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:49.296486 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hhl44" event={"ID":"352f238d-e5f0-4530-8b2e-63f15ab29644","Type":"ContainerStarted","Data":"8b2e9b694dd687eab76e535dc9d6cd63ffdf828e3f9eb28ce393488bec83a67b"} Apr 16 16:12:49.296524 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:49.296527 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hhl44" event={"ID":"352f238d-e5f0-4530-8b2e-63f15ab29644","Type":"ContainerStarted","Data":"8719c24dab03b86abc8bfc279c366a4fbc7e2e32e79e104e37c97a318760f5b0"} Apr 16 16:12:49.313263 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:49.313218 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-hhl44" podStartSLOduration=1.313203883 podStartE2EDuration="1.313203883s" podCreationTimestamp="2026-04-16 16:12:48 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 16:12:49.311121844 +0000 UTC m=+607.569868657" watchObservedRunningTime="2026-04-16 16:12:49.313203883 +0000 UTC m=+607.571950693" Apr 16 16:12:53.311343 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:53.311309 2576 generic.go:358] "Generic (PLEG): container finished" podID="352f238d-e5f0-4530-8b2e-63f15ab29644" containerID="8b2e9b694dd687eab76e535dc9d6cd63ffdf828e3f9eb28ce393488bec83a67b" exitCode=0 Apr 16 16:12:53.311709 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:53.311380 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hhl44" event={"ID":"352f238d-e5f0-4530-8b2e-63f15ab29644","Type":"ContainerDied","Data":"8b2e9b694dd687eab76e535dc9d6cd63ffdf828e3f9eb28ce393488bec83a67b"} Apr 16 16:12:54.433576 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:54.433553 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hhl44" Apr 16 16:12:54.468010 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:54.467986 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zlg8j\" (UniqueName: \"kubernetes.io/projected/352f238d-e5f0-4530-8b2e-63f15ab29644-kube-api-access-zlg8j\") pod \"352f238d-e5f0-4530-8b2e-63f15ab29644\" (UID: \"352f238d-e5f0-4530-8b2e-63f15ab29644\") " Apr 16 16:12:54.469925 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:54.469903 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352f238d-e5f0-4530-8b2e-63f15ab29644-kube-api-access-zlg8j" (OuterVolumeSpecName: "kube-api-access-zlg8j") pod "352f238d-e5f0-4530-8b2e-63f15ab29644" (UID: "352f238d-e5f0-4530-8b2e-63f15ab29644"). InnerVolumeSpecName "kube-api-access-zlg8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 16:12:54.569380 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:54.569315 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlg8j\" (UniqueName: \"kubernetes.io/projected/352f238d-e5f0-4530-8b2e-63f15ab29644-kube-api-access-zlg8j\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 16:12:55.318804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:55.318770 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-hhl44" event={"ID":"352f238d-e5f0-4530-8b2e-63f15ab29644","Type":"ContainerDied","Data":"8719c24dab03b86abc8bfc279c366a4fbc7e2e32e79e104e37c97a318760f5b0"} Apr 16 16:12:55.318804 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:55.318805 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8719c24dab03b86abc8bfc279c366a4fbc7e2e32e79e104e37c97a318760f5b0" Apr 16 16:12:55.319006 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:12:55.318815 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-hhl44" Apr 16 16:17:42.375420 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:17:42.375388 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:17:42.377403 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:17:42.377379 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:17:42.380025 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:17:42.379989 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:17:42.381858 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:17:42.381835 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:22:42.401147 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:22:42.401117 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:22:42.402611 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:22:42.402589 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:22:42.405761 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:22:42.405741 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:22:42.407236 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:22:42.407218 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:27:42.425747 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:27:42.425716 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:27:42.428216 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:27:42.428192 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:27:42.430490 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:27:42.430472 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:27:42.435081 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:27:42.435064 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:32:42.451342 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:32:42.451311 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:32:42.455554 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:32:42.455533 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:32:42.455742 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:32:42.455725 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:32:42.460189 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:32:42.460170 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:37:42.475668 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:37:42.475640 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:37:42.482562 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:37:42.482541 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:37:42.483593 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:37:42.483572 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:37:42.488785 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:37:42.488768 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:42:42.502634 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:42:42.502520 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:42:42.513539 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:42:42.513516 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:42:42.518738 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:42:42.518717 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:42:42.523086 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:42:42.523069 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:47:42.539581 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:47:42.539482 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:47:42.544032 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:47:42.543991 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:47:42.545067 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:47:42.545045 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:47:42.549029 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:47:42.548996 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:52:42.563539 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:52:42.563433 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:52:42.567867 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:52:42.567847 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:52:42.568730 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:52:42.568709 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:52:42.573148 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:52:42.573133 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:57:42.587932 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:57:42.587826 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:57:42.592319 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:57:42.592300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 16:57:42.593809 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:57:42.593790 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 16:57:42.597734 ip-10-0-143-177 kubenswrapper[2576]: I0416 16:57:42.597719 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 17:02:42.612272 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:02:42.612166 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 17:02:42.619076 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:02:42.616640 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 17:02:42.619076 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:02:42.617218 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 17:02:42.621499 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:02:42.621482 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 17:04:44.925328 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:44.925297 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d2k2c/must-gather-2zjpq"] Apr 16 17:04:44.925778 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:44.925632 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="352f238d-e5f0-4530-8b2e-63f15ab29644" containerName="s3-tls-init-serving" Apr 16 17:04:44.925778 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:44.925643 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="352f238d-e5f0-4530-8b2e-63f15ab29644" containerName="s3-tls-init-serving" Apr 16 17:04:44.925778 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:44.925702 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="352f238d-e5f0-4530-8b2e-63f15ab29644" containerName="s3-tls-init-serving" Apr 16 17:04:44.928947 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:44.928927 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" Apr 16 17:04:44.931235 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:44.931209 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d2k2c\"/\"default-dockercfg-6tvdv\"" Apr 16 17:04:44.931700 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:44.931676 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d2k2c\"/\"kube-root-ca.crt\"" Apr 16 17:04:44.931791 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:44.931714 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d2k2c\"/\"openshift-service-ca.crt\"" Apr 16 17:04:44.938230 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:44.938205 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d2k2c/must-gather-2zjpq"] Apr 16 17:04:45.097950 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:45.097917 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj9n5\" (UniqueName: \"kubernetes.io/projected/bb158784-f170-4011-b193-626c18e48bf9-kube-api-access-kj9n5\") pod \"must-gather-2zjpq\" (UID: \"bb158784-f170-4011-b193-626c18e48bf9\") " pod="openshift-must-gather-d2k2c/must-gather-2zjpq" Apr 16 17:04:45.098139 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:45.097963 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb158784-f170-4011-b193-626c18e48bf9-must-gather-output\") pod \"must-gather-2zjpq\" (UID: \"bb158784-f170-4011-b193-626c18e48bf9\") " pod="openshift-must-gather-d2k2c/must-gather-2zjpq" Apr 16 17:04:45.199257 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:45.199160 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb158784-f170-4011-b193-626c18e48bf9-must-gather-output\") pod \"must-gather-2zjpq\" (UID: \"bb158784-f170-4011-b193-626c18e48bf9\") " pod="openshift-must-gather-d2k2c/must-gather-2zjpq" Apr 16 17:04:45.199422 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:45.199298 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kj9n5\" (UniqueName: \"kubernetes.io/projected/bb158784-f170-4011-b193-626c18e48bf9-kube-api-access-kj9n5\") pod \"must-gather-2zjpq\" (UID: \"bb158784-f170-4011-b193-626c18e48bf9\") " pod="openshift-must-gather-d2k2c/must-gather-2zjpq" Apr 16 17:04:45.199517 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:45.199499 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb158784-f170-4011-b193-626c18e48bf9-must-gather-output\") pod \"must-gather-2zjpq\" (UID: \"bb158784-f170-4011-b193-626c18e48bf9\") " pod="openshift-must-gather-d2k2c/must-gather-2zjpq" Apr 16 17:04:45.211921 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:45.211869 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj9n5\" (UniqueName: \"kubernetes.io/projected/bb158784-f170-4011-b193-626c18e48bf9-kube-api-access-kj9n5\") pod \"must-gather-2zjpq\" (UID: \"bb158784-f170-4011-b193-626c18e48bf9\") " pod="openshift-must-gather-d2k2c/must-gather-2zjpq" Apr 16 17:04:45.250767 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:45.250733 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" Apr 16 17:04:45.368636 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:45.368611 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d2k2c/must-gather-2zjpq"] Apr 16 17:04:45.370674 ip-10-0-143-177 kubenswrapper[2576]: W0416 17:04:45.370641 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb158784_f170_4011_b193_626c18e48bf9.slice/crio-33dcb4b8520cb8037c017957e18862432a796af9e59b007e7a056603cc6fdc62 WatchSource:0}: Error finding container 33dcb4b8520cb8037c017957e18862432a796af9e59b007e7a056603cc6fdc62: Status 404 returned error can't find the container with id 33dcb4b8520cb8037c017957e18862432a796af9e59b007e7a056603cc6fdc62 Apr 16 17:04:45.372513 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:45.372488 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Apr 16 17:04:45.912239 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:45.912195 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" event={"ID":"bb158784-f170-4011-b193-626c18e48bf9","Type":"ContainerStarted","Data":"33dcb4b8520cb8037c017957e18862432a796af9e59b007e7a056603cc6fdc62"} Apr 16 17:04:49.932368 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:49.932283 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" event={"ID":"bb158784-f170-4011-b193-626c18e48bf9","Type":"ContainerStarted","Data":"881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842"} Apr 16 17:04:49.932368 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:04:49.932321 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" event={"ID":"bb158784-f170-4011-b193-626c18e48bf9","Type":"ContainerStarted","Data":"f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba"} Apr 16 17:05:09.006228 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:09.006197 2576 generic.go:358] "Generic (PLEG): container finished" podID="bb158784-f170-4011-b193-626c18e48bf9" containerID="f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba" exitCode=0 Apr 16 17:05:09.006641 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:09.006272 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" event={"ID":"bb158784-f170-4011-b193-626c18e48bf9","Type":"ContainerDied","Data":"f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba"} Apr 16 17:05:09.006641 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:09.006596 2576 scope.go:117] "RemoveContainer" containerID="f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba" Apr 16 17:05:09.847480 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:09.847449 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-d2k2c_must-gather-2zjpq_bb158784-f170-4011-b193-626c18e48bf9/gather/0.log" Apr 16 17:05:13.265733 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:13.265690 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cpt7l_9b57199f-943e-4015-b87b-a582c2c06494/global-pull-secret-syncer/0.log" Apr 16 17:05:13.585885 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:13.585809 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lg294_b5460383-84d2-4bd9-aa8b-82bbace94cb8/konnectivity-agent/0.log" Apr 16 17:05:13.769487 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:13.769459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-177.ec2.internal_a6c9c3cf395cb0ad946e6c0c60278a83/haproxy/0.log" Apr 16 17:05:15.313000 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.312954 2576 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-d2k2c/must-gather-2zjpq"] Apr 16 17:05:15.313520 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.313255 2576 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" podUID="bb158784-f170-4011-b193-626c18e48bf9" containerName="copy" containerID="cri-o://881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842" gracePeriod=2 Apr 16 17:05:15.315735 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.315701 2576 status_manager.go:895] "Failed to get status for pod" podUID="bb158784-f170-4011-b193-626c18e48bf9" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" err="pods \"must-gather-2zjpq\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-d2k2c\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" Apr 16 17:05:15.316566 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.316539 2576 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-d2k2c/must-gather-2zjpq"] Apr 16 17:05:15.536484 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.536459 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-d2k2c_must-gather-2zjpq_bb158784-f170-4011-b193-626c18e48bf9/copy/0.log" Apr 16 17:05:15.536803 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.536791 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" Apr 16 17:05:15.538861 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.538837 2576 status_manager.go:895] "Failed to get status for pod" podUID="bb158784-f170-4011-b193-626c18e48bf9" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" err="pods \"must-gather-2zjpq\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-d2k2c\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" Apr 16 17:05:15.687115 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.687003 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb158784-f170-4011-b193-626c18e48bf9-must-gather-output\") pod \"bb158784-f170-4011-b193-626c18e48bf9\" (UID: \"bb158784-f170-4011-b193-626c18e48bf9\") " Apr 16 17:05:15.687115 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.687065 2576 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj9n5\" (UniqueName: \"kubernetes.io/projected/bb158784-f170-4011-b193-626c18e48bf9-kube-api-access-kj9n5\") pod \"bb158784-f170-4011-b193-626c18e48bf9\" (UID: \"bb158784-f170-4011-b193-626c18e48bf9\") " Apr 16 17:05:15.688395 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.688356 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb158784-f170-4011-b193-626c18e48bf9-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "bb158784-f170-4011-b193-626c18e48bf9" (UID: "bb158784-f170-4011-b193-626c18e48bf9"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Apr 16 17:05:15.689246 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.689227 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb158784-f170-4011-b193-626c18e48bf9-kube-api-access-kj9n5" (OuterVolumeSpecName: "kube-api-access-kj9n5") pod "bb158784-f170-4011-b193-626c18e48bf9" (UID: "bb158784-f170-4011-b193-626c18e48bf9"). InnerVolumeSpecName "kube-api-access-kj9n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Apr 16 17:05:15.788084 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.788050 2576 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/bb158784-f170-4011-b193-626c18e48bf9-must-gather-output\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 17:05:15.788084 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:15.788079 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj9n5\" (UniqueName: \"kubernetes.io/projected/bb158784-f170-4011-b193-626c18e48bf9-kube-api-access-kj9n5\") on node \"ip-10-0-143-177.ec2.internal\" DevicePath \"\"" Apr 16 17:05:16.030729 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.030704 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-d2k2c_must-gather-2zjpq_bb158784-f170-4011-b193-626c18e48bf9/copy/0.log" Apr 16 17:05:16.031047 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.031006 2576 generic.go:358] "Generic (PLEG): container finished" podID="bb158784-f170-4011-b193-626c18e48bf9" containerID="881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842" exitCode=143 Apr 16 17:05:16.031120 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.031082 2576 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" Apr 16 17:05:16.031161 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.031122 2576 scope.go:117] "RemoveContainer" containerID="881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842" Apr 16 17:05:16.033548 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.033521 2576 status_manager.go:895] "Failed to get status for pod" podUID="bb158784-f170-4011-b193-626c18e48bf9" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" err="pods \"must-gather-2zjpq\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-d2k2c\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" Apr 16 17:05:16.039083 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.039062 2576 scope.go:117] "RemoveContainer" containerID="f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba" Apr 16 17:05:16.041358 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.041334 2576 status_manager.go:895] "Failed to get status for pod" podUID="bb158784-f170-4011-b193-626c18e48bf9" pod="openshift-must-gather-d2k2c/must-gather-2zjpq" err="pods \"must-gather-2zjpq\" is forbidden: User \"system:node:ip-10-0-143-177.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-d2k2c\": no relationship found between node 'ip-10-0-143-177.ec2.internal' and this object" Apr 16 17:05:16.050824 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.050808 2576 scope.go:117] "RemoveContainer" containerID="881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842" Apr 16 17:05:16.051114 ip-10-0-143-177 kubenswrapper[2576]: E0416 17:05:16.051089 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842\": container with ID starting with 881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842 not found: ID does not exist" containerID="881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842" Apr 16 17:05:16.051175 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.051123 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842"} err="failed to get container status \"881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842\": rpc error: code = NotFound desc = could not find container \"881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842\": container with ID starting with 881dd5f27ebe73529c80b38bb349a06ac15dab86e4bd23f71107719325362842 not found: ID does not exist" Apr 16 17:05:16.051175 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.051140 2576 scope.go:117] "RemoveContainer" containerID="f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba" Apr 16 17:05:16.051387 ip-10-0-143-177 kubenswrapper[2576]: E0416 17:05:16.051370 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba\": container with ID starting with f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba not found: ID does not exist" containerID="f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba" Apr 16 17:05:16.051421 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.051393 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba"} err="failed to get container status \"f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba\": rpc error: code = NotFound desc = could not find container \"f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba\": container with ID starting with f3d0106fee72cedcdc192c8b04726a5ee415de3746725ab248c09efc0f8116ba not found: ID does not exist" Apr 16 17:05:16.418272 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.418199 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb158784-f170-4011-b193-626c18e48bf9" path="/var/lib/kubelet/pods/bb158784-f170-4011-b193-626c18e48bf9/volumes" Apr 16 17:05:16.516699 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.516656 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25334c8e-c613-4d85-848f-f5c98c784a0c/alertmanager/0.log" Apr 16 17:05:16.552852 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.552826 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25334c8e-c613-4d85-848f-f5c98c784a0c/config-reloader/0.log" Apr 16 17:05:16.589779 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.589752 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25334c8e-c613-4d85-848f-f5c98c784a0c/kube-rbac-proxy-web/0.log" Apr 16 17:05:16.622448 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.622425 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25334c8e-c613-4d85-848f-f5c98c784a0c/kube-rbac-proxy/0.log" Apr 16 17:05:16.656600 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.656574 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25334c8e-c613-4d85-848f-f5c98c784a0c/kube-rbac-proxy-metric/0.log" Apr 16 17:05:16.687761 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.687698 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25334c8e-c613-4d85-848f-f5c98c784a0c/prom-label-proxy/0.log" Apr 16 17:05:16.711601 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.711569 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_25334c8e-c613-4d85-848f-f5c98c784a0c/init-config-reloader/0.log" Apr 16 17:05:16.763414 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.763387 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6667474d89-w6kz2_7d332411-d30c-4247-a333-0a0f77355443/cluster-monitoring-operator/0.log" Apr 16 17:05:16.894955 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.894926 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6bbf66bb88-nl6hh_60dee853-2b4a-44c9-a555-91259c774dbe/metrics-server/0.log" Apr 16 17:05:16.969303 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.969277 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6fv2w_8756041d-afbd-4507-a82f-426bddd41acb/node-exporter/0.log" Apr 16 17:05:16.995652 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:16.995631 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6fv2w_8756041d-afbd-4507-a82f-426bddd41acb/kube-rbac-proxy/0.log" Apr 16 17:05:17.025354 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.025335 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6fv2w_8756041d-afbd-4507-a82f-426bddd41acb/init-textfile/0.log" Apr 16 17:05:17.328509 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.328432 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a6123c0-9be7-4f39-be1c-f5b766e57efd/prometheus/0.log" Apr 16 17:05:17.346692 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.346664 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a6123c0-9be7-4f39-be1c-f5b766e57efd/config-reloader/0.log" Apr 16 17:05:17.369061 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.369036 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a6123c0-9be7-4f39-be1c-f5b766e57efd/thanos-sidecar/0.log" Apr 16 17:05:17.392197 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.392174 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a6123c0-9be7-4f39-be1c-f5b766e57efd/kube-rbac-proxy-web/0.log" Apr 16 17:05:17.418674 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.418647 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a6123c0-9be7-4f39-be1c-f5b766e57efd/kube-rbac-proxy/0.log" Apr 16 17:05:17.441321 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.441292 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a6123c0-9be7-4f39-be1c-f5b766e57efd/kube-rbac-proxy-thanos/0.log" Apr 16 17:05:17.469639 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.469608 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9a6123c0-9be7-4f39-be1c-f5b766e57efd/init-config-reloader/0.log" Apr 16 17:05:17.545789 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.545763 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-9cb97cd87-nwx8h_2891f287-27d8-4b90-b731-016769f80008/prometheus-operator-admission-webhook/0.log" Apr 16 17:05:17.575008 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.574983 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-566567f95c-vxvnr_7b2531bf-9760-40c2-9a24-03df27efff16/telemeter-client/0.log" Apr 16 17:05:17.599282 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.599217 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-566567f95c-vxvnr_7b2531bf-9760-40c2-9a24-03df27efff16/reload/0.log" Apr 16 17:05:17.623427 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.623405 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-566567f95c-vxvnr_7b2531bf-9760-40c2-9a24-03df27efff16/kube-rbac-proxy/0.log" Apr 16 17:05:17.656375 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.656351 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cdb848f8b-zmtdr_3fc25446-af98-45ae-bf8c-d7747982bc50/thanos-query/0.log" Apr 16 17:05:17.676443 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.676415 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cdb848f8b-zmtdr_3fc25446-af98-45ae-bf8c-d7747982bc50/kube-rbac-proxy-web/0.log" Apr 16 17:05:17.695953 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.695933 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cdb848f8b-zmtdr_3fc25446-af98-45ae-bf8c-d7747982bc50/kube-rbac-proxy/0.log" Apr 16 17:05:17.717183 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.717165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cdb848f8b-zmtdr_3fc25446-af98-45ae-bf8c-d7747982bc50/prom-label-proxy/0.log" Apr 16 17:05:17.739899 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.739882 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cdb848f8b-zmtdr_3fc25446-af98-45ae-bf8c-d7747982bc50/kube-rbac-proxy-rules/0.log" Apr 16 17:05:17.764231 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:17.764209 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6cdb848f8b-zmtdr_3fc25446-af98-45ae-bf8c-d7747982bc50/kube-rbac-proxy-metrics/0.log" Apr 16 17:05:18.946386 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:18.946352 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5cb6cf4cb4-j6j9c_5e3c6810-602d-4451-a18d-2b7f8b3e6245/networking-console-plugin/0.log" Apr 16 17:05:19.414500 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:19.414471 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/2.log" Apr 16 17:05:19.421445 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:19.421417 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-d87b8d5fc-qb2g6_9541790d-c119-472d-ab6a-4915d4065240/console-operator/3.log" Apr 16 17:05:20.237975 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.237947 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-7d955d5dd4-k947k_0c553c40-80b0-4906-9c86-e744ca5c2eba/volume-data-source-validator/0.log" Apr 16 17:05:20.933478 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.933450 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mt4rq_ad7697a0-8cda-49df-861c-5762c30a7794/dns/0.log" Apr 16 17:05:20.961048 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.961006 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mt4rq_ad7697a0-8cda-49df-861c-5762c30a7794/kube-rbac-proxy/0.log" Apr 16 17:05:20.969351 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.969328 2576 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz"] Apr 16 17:05:20.969694 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.969681 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb158784-f170-4011-b193-626c18e48bf9" containerName="copy" Apr 16 17:05:20.969741 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.969696 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb158784-f170-4011-b193-626c18e48bf9" containerName="copy" Apr 16 17:05:20.969741 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.969707 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb158784-f170-4011-b193-626c18e48bf9" containerName="gather" Apr 16 17:05:20.969741 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.969713 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb158784-f170-4011-b193-626c18e48bf9" containerName="gather" Apr 16 17:05:20.969833 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.969764 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb158784-f170-4011-b193-626c18e48bf9" containerName="copy" Apr 16 17:05:20.969833 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.969772 2576 memory_manager.go:356] "RemoveStaleState removing state" podUID="bb158784-f170-4011-b193-626c18e48bf9" containerName="gather" Apr 16 17:05:20.973190 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.973172 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:20.975568 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.975544 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7k2bn\"/\"openshift-service-ca.crt\"" Apr 16 17:05:20.975568 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.975564 2576 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7k2bn\"/\"kube-root-ca.crt\"" Apr 16 17:05:20.975717 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.975604 2576 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7k2bn\"/\"default-dockercfg-nkv5v\"" Apr 16 17:05:20.978843 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:20.978823 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz"] Apr 16 17:05:21.084812 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.084784 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-klzvx_dd45d1df-4307-46b1-878a-6073d5e56455/dns-node-resolver/0.log" Apr 16 17:05:21.141265 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.141234 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-podres\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.141468 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.141283 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-proc\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.141468 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.141372 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5cf\" (UniqueName: \"kubernetes.io/projected/32b7435b-e625-4bc8-b71d-39ab0515474f-kube-api-access-ft5cf\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.141468 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.141410 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-sys\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.141468 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.141456 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-lib-modules\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.242654 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.242621 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-podres\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.243109 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.242672 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-proc\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.243109 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.242707 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5cf\" (UniqueName: \"kubernetes.io/projected/32b7435b-e625-4bc8-b71d-39ab0515474f-kube-api-access-ft5cf\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.243109 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.242724 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-sys\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.243109 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.242750 2576 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-lib-modules\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.243109 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.242822 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-podres\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.243109 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.242842 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-proc\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.243109 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.242888 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-sys\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.243109 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.242957 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32b7435b-e625-4bc8-b71d-39ab0515474f-lib-modules\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.251227 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.251200 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5cf\" (UniqueName: \"kubernetes.io/projected/32b7435b-e625-4bc8-b71d-39ab0515474f-kube-api-access-ft5cf\") pod \"perf-node-gather-daemonset-8zmqz\" (UID: \"32b7435b-e625-4bc8-b71d-39ab0515474f\") " pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.284456 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.284427 2576 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:21.554464 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.554351 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bgdhp_0020ce4c-790d-41fd-9e3a-6ada38675cda/node-ca/0.log" Apr 16 17:05:21.613145 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:21.613070 2576 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz"] Apr 16 17:05:21.615356 ip-10-0-143-177 kubenswrapper[2576]: W0416 17:05:21.615331 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod32b7435b_e625_4bc8_b71d_39ab0515474f.slice/crio-617861b021ff75fa9a1f87e7ea47c10984e2971d356d9269192acc30a700d9f6 WatchSource:0}: Error finding container 617861b021ff75fa9a1f87e7ea47c10984e2971d356d9269192acc30a700d9f6: Status 404 returned error can't find the container with id 617861b021ff75fa9a1f87e7ea47c10984e2971d356d9269192acc30a700d9f6 Apr 16 17:05:22.050632 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:22.050591 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" event={"ID":"32b7435b-e625-4bc8-b71d-39ab0515474f","Type":"ContainerStarted","Data":"57296dc7e5e229851d9f1dfafe73ca629227b2038dd068c0e99a5539c1ce1a04"} Apr 16 17:05:22.050632 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:22.050632 2576 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" event={"ID":"32b7435b-e625-4bc8-b71d-39ab0515474f","Type":"ContainerStarted","Data":"617861b021ff75fa9a1f87e7ea47c10984e2971d356d9269192acc30a700d9f6"} Apr 16 17:05:22.050881 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:22.050709 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:22.068600 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:22.068553 2576 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" podStartSLOduration=2.068536989 podStartE2EDuration="2.068536989s" podCreationTimestamp="2026-04-16 17:05:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-04-16 17:05:22.067952887 +0000 UTC m=+3760.326699697" watchObservedRunningTime="2026-04-16 17:05:22.068536989 +0000 UTC m=+3760.327283799" Apr 16 17:05:22.258296 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:22.258265 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6fdcb8c7b8-jprpz_66494738-0f0c-42ff-b8e6-c0961cc5f3c1/router/0.log" Apr 16 17:05:22.586125 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:22.586094 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kmvkl_f4bb9693-c1c3-4ecc-b82d-2ea9913caa85/serve-healthcheck-canary/0.log" Apr 16 17:05:22.941901 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:22.941816 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5785d4fcdd-q2snw_dbf86cc5-0e80-47e9-b081-8510a5c57973/insights-operator/0.log" Apr 16 17:05:22.943164 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:22.943144 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5785d4fcdd-q2snw_dbf86cc5-0e80-47e9-b081-8510a5c57973/insights-operator/1.log" Apr 16 17:05:23.089862 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:23.089833 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gpd5d_17a67b97-ec02-466c-9534-4b23a9e0997e/kube-rbac-proxy/0.log" Apr 16 17:05:23.109002 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:23.108973 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gpd5d_17a67b97-ec02-466c-9534-4b23a9e0997e/exporter/0.log" Apr 16 17:05:23.133370 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:23.133337 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gpd5d_17a67b97-ec02-466c-9534-4b23a9e0997e/extractor/0.log" Apr 16 17:05:25.408305 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:25.408278 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hl4hx_1527e1d7-a803-43d1-bcee-05d26181772f/s3-init/0.log" Apr 16 17:05:25.430939 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:25.430908 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-vtd2d_85284f39-75fd-460f-85f4-5d2798acc482/s3-tls-init-custom/0.log" Apr 16 17:05:25.454503 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:25.454475 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-hhl44_352f238d-e5f0-4530-8b2e-63f15ab29644/s3-tls-init-serving/0.log" Apr 16 17:05:28.064498 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:28.064472 2576 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7k2bn/perf-node-gather-daemonset-8zmqz" Apr 16 17:05:29.461581 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:29.461547 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-756bb7d76f-htwnr_057a2387-92a7-47ee-8fbd-ff2d0c13ee1a/kube-storage-version-migrator-operator/1.log" Apr 16 17:05:29.463525 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:29.463498 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-756bb7d76f-htwnr_057a2387-92a7-47ee-8fbd-ff2d0c13ee1a/kube-storage-version-migrator-operator/0.log" Apr 16 17:05:30.516213 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:30.516124 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-68jlg_3a79498a-1173-46d9-921c-ebdf5b7ece1f/kube-multus/0.log" Apr 16 17:05:30.702708 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:30.702676 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dfnlw_4bb6679c-eb46-4326-bf8e-30829f808afc/kube-multus-additional-cni-plugins/0.log" Apr 16 17:05:30.727574 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:30.727544 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dfnlw_4bb6679c-eb46-4326-bf8e-30829f808afc/egress-router-binary-copy/0.log" Apr 16 17:05:30.752405 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:30.752377 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dfnlw_4bb6679c-eb46-4326-bf8e-30829f808afc/cni-plugins/0.log" Apr 16 17:05:30.771697 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:30.771635 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dfnlw_4bb6679c-eb46-4326-bf8e-30829f808afc/bond-cni-plugin/0.log" Apr 16 17:05:30.795690 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:30.795665 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dfnlw_4bb6679c-eb46-4326-bf8e-30829f808afc/routeoverride-cni/0.log" Apr 16 17:05:30.819142 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:30.819117 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dfnlw_4bb6679c-eb46-4326-bf8e-30829f808afc/whereabouts-cni-bincopy/0.log" Apr 16 17:05:30.842473 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:30.842445 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dfnlw_4bb6679c-eb46-4326-bf8e-30829f808afc/whereabouts-cni/0.log" Apr 16 17:05:31.130329 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:31.130244 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fpztb_cd5f8939-2604-4ca4-8d1e-05dc7f86fa41/network-metrics-daemon/0.log" Apr 16 17:05:31.148387 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:31.148357 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-fpztb_cd5f8939-2604-4ca4-8d1e-05dc7f86fa41/kube-rbac-proxy/0.log" Apr 16 17:05:32.203989 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:32.203956 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-controller/0.log" Apr 16 17:05:32.220258 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:32.220232 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/0.log" Apr 16 17:05:32.254880 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:32.254828 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovn-acl-logging/1.log" Apr 16 17:05:32.281125 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:32.281092 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/kube-rbac-proxy-node/0.log" Apr 16 17:05:32.304764 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:32.304723 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/kube-rbac-proxy-ovn-metrics/0.log" Apr 16 17:05:32.326398 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:32.326364 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/northd/0.log" Apr 16 17:05:32.347550 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:32.347523 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/nbdb/0.log" Apr 16 17:05:32.370864 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:32.370833 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/sbdb/0.log" Apr 16 17:05:32.542745 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:32.542713 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gtjzw_c78e2321-0a40-4d26-89b3-afd2d6eee0a6/ovnkube-controller/0.log" Apr 16 17:05:33.744423 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:33.744393 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-2lh6g_23dd67cf-2731-4685-80f7-f4a1e75b6476/network-check-target-container/0.log" Apr 16 17:05:34.705282 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:34.705247 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-m98lv_d44ec06b-81b1-4bac-9abd-8efadd98b94a/iptables-alerter/0.log" Apr 16 17:05:35.355258 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:35.355222 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-q5jgq_4c0f9a7c-7995-43e1-9141-56467d871fce/tuned/0.log" Apr 16 17:05:37.064382 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:37.064347 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-667775844f-28sdj_25e38cb7-4445-4ec1-8afb-75356dc1c085/cluster-samples-operator/0.log" Apr 16 17:05:37.080605 ip-10-0-143-177 kubenswrapper[2576]: I0416 17:05:37.080575 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-667775844f-28sdj_25e38cb7-4445-4ec1-8afb-75356dc1c085/cluster-samples-operator-watch/0.log"