W0311 04:15:34.748461 1 cmd.go:257] Using insecure, self-signed certificates I0311 04:15:34.962114 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0311 04:15:34.962449 1 observer_polling.go:159] Starting file observer W0311 04:15:34.985048 1 builder.go:272] unable to get owner reference (falling back to namespace): replicasets.apps "insights-operator-5ff5cb4f99" is forbidden: User "system:serviceaccount:openshift-insights:operator" cannot get resource "replicasets" in API group "apps" in the namespace "openshift-insights" I0311 04:15:35.375894 1 operator.go:59] Starting insights-operator v0.0.0-master+$Format:%H$ I0311 04:15:35.376107 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0311 04:15:35.376680 1 secure_serving.go:57] Forcing use of http/1.1 only W0311 04:15:35.376701 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0311 04:15:35.376706 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0311 04:15:35.376710 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0311 04:15:35.376714 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0311 04:15:35.376717 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0311 04:15:35.376719 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0311 04:15:35.376829 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0311 04:15:35.380099 1 operator.go:124] FeatureGates initialized: knownFeatureGates=[AWSEFSDriverVolumeMetrics AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BareMetalLoadBalancer BuildCSIVolumes CPMSMachineNamePrefix ChunkSizeMiB CloudDualStackNodeIPs ConsolePluginContentSecurityPolicy DisableKubeletCloudCredentialProviders ExternalOIDC GCPLabelsTags GatewayAPI GatewayAPIController HardwareSpeed IngressControllerLBSubnetsAWS KMSv1 ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles MultiArchInstallAWS MultiArchInstallGCP NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation NodeDisruptionPolicy OnClusterBuild PersistentIPsForVirtualization PrivateHostedZoneAWS RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController VSphereDriverConfiguration VSphereMultiVCenters ValidatingAdmissionPolicy AWSClusterHostedDNS AutomatedEtcdBackup BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNS GCPCustomAPIEndpoints HighlyAvailableArbiter ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather InsightsRuntimeExtractor KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController MachineAPIProviderOpenStack MachineConfigNodes MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PinnedImages PlatformOperators ProcMountType RouteAdvertisements SELinuxChangePolicy SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerification SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass VolumeGroupSnapshot] I0311 04:15:35.380186 1 event.go:377] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"openshift-insights", Name:"openshift-insights", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AWSEFSDriverVolumeMetrics", "AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BareMetalLoadBalancer", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ChunkSizeMiB", "CloudDualStackNodeIPs", "ConsolePluginContentSecurityPolicy", "DisableKubeletCloudCredentialProviders", "ExternalOIDC", "GCPLabelsTags", "GatewayAPI", "GatewayAPIController", "HardwareSpeed", "IngressControllerLBSubnetsAWS", "KMSv1", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "MultiArchInstallAWS", "MultiArchInstallGCP", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "NodeDisruptionPolicy", "OnClusterBuild", "PersistentIPsForVirtualization", "PrivateHostedZoneAWS", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "VSphereDriverConfiguration", "VSphereMultiVCenters", "ValidatingAdmissionPolicy"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AutomatedEtcdBackup", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "HighlyAvailableArbiter", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "InsightsRuntimeExtractor", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "MachineAPIProviderOpenStack", "MachineConfigNodes", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PinnedImages", "PlatformOperators", "ProcMountType", "RouteAdvertisements", "SELinuxChangePolicy", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0311 04:15:35.381682 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0311 04:15:35.381704 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0311 04:15:35.381757 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0311 04:15:35.381781 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0311 04:15:35.381757 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0311 04:15:35.381801 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0311 04:15:35.382072 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-3822289155/tls.crt::/tmp/serving-cert-3822289155/tls.key" I0311 04:15:35.382226 1 secure_serving.go:213] Serving securely on [::]:8443 I0311 04:15:35.382242 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" E0311 04:15:35.382864 1 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:serviceaccount:openshift-insights:operator\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-insights\"" event="&Event{ObjectMeta:{openshift-insights.189bae4fa426b9a5 openshift-insights 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Namespace,Namespace:openshift-insights,Name:openshift-insights,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:FeatureGatesInitialized,Message:FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{\"AWSEFSDriverVolumeMetrics\", \"AdditionalRoutingCapabilities\", \"AdminNetworkPolicy\", \"AlibabaPlatform\", \"AzureWorkloadIdentity\", \"BareMetalLoadBalancer\", \"BuildCSIVolumes\", \"CPMSMachineNamePrefix\", \"ChunkSizeMiB\", \"CloudDualStackNodeIPs\", \"ConsolePluginContentSecurityPolicy\", \"DisableKubeletCloudCredentialProviders\", \"ExternalOIDC\", \"GCPLabelsTags\", \"GatewayAPI\", \"GatewayAPIController\", \"HardwareSpeed\", \"IngressControllerLBSubnetsAWS\", \"KMSv1\", \"ManagedBootImages\", \"ManagedBootImagesAWS\", \"MetricsCollectionProfiles\", \"MultiArchInstallAWS\", \"MultiArchInstallGCP\", \"NetworkDiagnosticsConfig\", \"NetworkLiveMigration\", \"NetworkSegmentation\", \"NodeDisruptionPolicy\", \"OnClusterBuild\", \"PersistentIPsForVirtualization\", \"PrivateHostedZoneAWS\", \"RouteExternalCertificate\", \"ServiceAccountTokenNodeBinding\", \"SetEIPForNLBIngressController\", \"VSphereDriverConfiguration\", \"VSphereMultiVCenters\", \"ValidatingAdmissionPolicy\"}, Disabled:[]v1.FeatureGateName{\"AWSClusterHostedDNS\", \"AutomatedEtcdBackup\", \"BootcNodeManagement\", \"ClusterAPIInstall\", \"ClusterAPIInstallIBMCloud\", \"ClusterMonitoringConfig\", \"ClusterVersionOperatorConfiguration\", \"DNSNameResolver\", \"DualReplica\", \"DyanmicServiceEndpointIBMCloud\", \"DynamicResourceAllocation\", \"EtcdBackendQuota\", \"EventedPLEG\", \"Example\", \"Example2\", \"ExternalOIDCWithUIDAndExtraClaimMappings\", \"GCPClusterHostedDNS\", \"GCPCustomAPIEndpoints\", \"HighlyAvailableArbiter\", \"ImageStreamImportMode\", \"IngressControllerDynamicConfigurationManager\", \"InsightsConfig\", \"InsightsConfigAPI\", \"InsightsOnDemandDataGather\", \"InsightsRuntimeExtractor\", \"KMSEncryptionProvider\", \"MachineAPIMigration\", \"MachineAPIOperatorDisableMachineHealthCheckController\", \"MachineAPIProviderOpenStack\", \"MachineConfigNodes\", \"MaxUnavailableStatefulSet\", \"MinimumKubeletVersion\", \"MixedCPUsAllocation\", \"MultiArchInstallAzure\", \"NewOLM\", \"NewOLMCatalogdAPIV1Metas\", \"NewOLMOwnSingleNamespace\", \"NewOLMPreflightPermissionChecks\", \"NodeSwap\", \"NutanixMultiSubnets\", \"OVNObservability\", \"OpenShiftPodSecurityAdmission\", \"PinnedImages\", \"PlatformOperators\", \"ProcMountType\", \"RouteAdvertisements\", \"SELinuxChangePolicy\", \"SELinuxMount\", \"ShortCertRotation\", \"SignatureStores\", \"SigstoreImageVerification\", \"SigstoreImageVerificationPKI\", \"TranslateStreamCloseWebsocketRequests\", \"UpgradeStatus\", \"UserNamespacesPodSecurityStandards\", \"UserNamespacesSupport\", \"VSphereConfigurableMaxAllowedBlockVolumesPerNode\", \"VSphereHostVMGroupZonal\", \"VSphereMultiDisk\", \"VSphereMultiNetworks\", \"VolumeAttributesClass\", \"VolumeGroupSnapshot\"}},Source:EventSource{Component:openshift-insights-operator,Host:,},FirstTimestamp:2026-03-11 04:15:35.380072869 +0000 UTC m=+0.672926114,LastTimestamp:2026-03-11 04:15:35.380072869 +0000 UTC m=+0.672926114,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:openshift-insights-operator,ReportingInstance:,}" W0311 04:15:35.386215 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0311 04:15:35.386250 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0311 04:15:35.386373 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0311 04:15:35.391378 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0311 04:15:35.391401 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0311 04:15:35.396091 1 secretconfigobserver.go:119] support secret does not exist I0311 04:15:35.401376 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0311 04:15:35.408126 1 secretconfigobserver.go:119] support secret does not exist I0311 04:15:35.410406 1 recorder.go:156] Pruning old reports every 8h27m57s, max age is 288h0m0s I0311 04:15:35.416115 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0311 04:15:35.416120 1 periodic.go:212] Running clusterconfig gatherer I0311 04:15:35.416136 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0311 04:15:35.416155 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0311 04:15:35.416160 1 insightsreport.go:296] Starting report retriever I0311 04:15:35.416163 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0311 04:15:35.416183 1 tasks_processing.go:45] number of workers: 32 I0311 04:15:35.416210 1 tasks_processing.go:69] worker 1 listening for tasks. I0311 04:15:35.416224 1 tasks_processing.go:69] worker 0 listening for tasks. I0311 04:15:35.416228 1 tasks_processing.go:69] worker 10 listening for tasks. I0311 04:15:35.416235 1 tasks_processing.go:69] worker 11 listening for tasks. I0311 04:15:35.416236 1 tasks_processing.go:71] worker 0 working on image_registries task. I0311 04:15:35.416242 1 tasks_processing.go:71] worker 11 working on image task. I0311 04:15:35.416243 1 tasks_processing.go:69] worker 6 listening for tasks. I0311 04:15:35.416251 1 tasks_processing.go:69] worker 7 listening for tasks. I0311 04:15:35.416250 1 tasks_processing.go:69] worker 5 listening for tasks. I0311 04:15:35.416259 1 tasks_processing.go:69] worker 14 listening for tasks. I0311 04:15:35.416261 1 tasks_processing.go:69] worker 8 listening for tasks. I0311 04:15:35.416268 1 tasks_processing.go:69] worker 9 listening for tasks. I0311 04:15:35.416269 1 tasks_processing.go:69] worker 4 listening for tasks. I0311 04:15:35.416230 1 tasks_processing.go:69] worker 17 listening for tasks. I0311 04:15:35.416275 1 tasks_processing.go:69] worker 12 listening for tasks. I0311 04:15:35.416281 1 tasks_processing.go:69] worker 25 listening for tasks. I0311 04:15:35.416282 1 tasks_processing.go:69] worker 13 listening for tasks. I0311 04:15:35.416287 1 tasks_processing.go:69] worker 19 listening for tasks. I0311 04:15:35.416287 1 tasks_processing.go:69] worker 3 listening for tasks. I0311 04:15:35.416290 1 tasks_processing.go:69] worker 22 listening for tasks. I0311 04:15:35.416295 1 tasks_processing.go:69] worker 2 listening for tasks. I0311 04:15:35.416295 1 tasks_processing.go:69] worker 18 listening for tasks. I0311 04:15:35.416300 1 tasks_processing.go:69] worker 21 listening for tasks. I0311 04:15:35.416305 1 tasks_processing.go:69] worker 24 listening for tasks. I0311 04:15:35.416307 1 tasks_processing.go:69] worker 20 listening for tasks. I0311 04:15:35.416298 1 tasks_processing.go:69] worker 23 listening for tasks. I0311 04:15:35.416307 1 tasks_processing.go:69] worker 29 listening for tasks. I0311 04:15:35.416312 1 tasks_processing.go:69] worker 15 listening for tasks. I0311 04:15:35.416315 1 tasks_processing.go:71] worker 1 working on openshift_machine_api_events task. I0311 04:15:35.416320 1 tasks_processing.go:69] worker 28 listening for tasks. I0311 04:15:35.416325 1 tasks_processing.go:69] worker 26 listening for tasks. I0311 04:15:35.416329 1 tasks_processing.go:71] worker 15 working on tsdb_status task. I0311 04:15:35.416335 1 tasks_processing.go:71] worker 13 working on schedulers task. I0311 04:15:35.416336 1 tasks_processing.go:71] worker 26 working on lokistack task. I0311 04:15:35.416323 1 tasks_processing.go:69] worker 30 listening for tasks. I0311 04:15:35.416351 1 tasks_processing.go:71] worker 30 working on silenced_alerts task. W0311 04:15:35.416357 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0311 04:15:35.416369 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 28.687µs to process 0 records I0311 04:15:35.416377 1 tasks_processing.go:71] worker 19 working on node_logs task. W0311 04:15:35.416379 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0311 04:15:35.416391 1 tasks_processing.go:71] worker 21 working on validating_webhook_configurations task. I0311 04:15:35.416396 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 31.446µs to process 0 records I0311 04:15:35.416236 1 tasks_processing.go:71] worker 10 working on cluster_apiserver task. I0311 04:15:35.416436 1 tasks_processing.go:71] worker 20 working on machine_configs task. I0311 04:15:35.416545 1 tasks_processing.go:71] worker 30 working on version task. I0311 04:15:35.416305 1 tasks_processing.go:69] worker 16 listening for tasks. I0311 04:15:35.416674 1 tasks_processing.go:71] worker 16 working on ceph_cluster task. I0311 04:15:35.416703 1 tasks_processing.go:71] worker 24 working on machine_sets task. I0311 04:15:35.416757 1 tasks_processing.go:71] worker 8 working on nodenetworkconfigurationpolicies task. I0311 04:15:35.416834 1 tasks_processing.go:71] worker 22 working on active_alerts task. I0311 04:15:35.416889 1 tasks_processing.go:71] worker 2 working on sap_pods task. I0311 04:15:35.416943 1 tasks_processing.go:71] worker 12 working on authentication task. I0311 04:15:35.416952 1 tasks_processing.go:71] worker 4 working on feature_gates task. I0311 04:15:35.416952 1 tasks_processing.go:71] worker 25 working on service_accounts task. W0311 04:15:35.416894 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0311 04:15:35.417219 1 tasks_processing.go:71] worker 22 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0311 04:15:35.416331 1 tasks_processing.go:71] worker 28 working on clusterroles task. I0311 04:15:35.416739 1 tasks_processing.go:71] worker 6 working on container_images task. I0311 04:15:35.416745 1 tasks_processing.go:71] worker 7 working on openstack_dataplanedeployments task. I0311 04:15:35.416749 1 tasks_processing.go:71] worker 5 working on monitoring_persistent_volumes task. I0311 04:15:35.416752 1 tasks_processing.go:71] worker 14 working on openshift_logging task. I0311 04:15:35.416905 1 tasks_processing.go:71] worker 3 working on nodes task. I0311 04:15:35.416317 1 tasks_processing.go:69] worker 27 listening for tasks. I0311 04:15:35.417866 1 tasks_processing.go:71] worker 27 working on ingress_certificates task. I0311 04:15:35.416315 1 tasks_processing.go:69] worker 31 listening for tasks. I0311 04:15:35.416920 1 tasks_processing.go:71] worker 23 working on pdbs task. I0311 04:15:35.417906 1 tasks_processing.go:71] worker 31 working on aggregated_monitoring_cr_names task. I0311 04:15:35.416929 1 tasks_processing.go:71] worker 15 working on operators_pods_and_events task. I0311 04:15:35.416932 1 tasks_processing.go:71] worker 18 working on openstack_dataplanenodesets task. I0311 04:15:35.416944 1 tasks_processing.go:71] worker 17 working on support_secret task. I0311 04:15:35.416940 1 tasks_processing.go:71] worker 29 working on machine_config_pools task. I0311 04:15:35.416949 1 tasks_processing.go:71] worker 9 working on openstack_controlplanes task. I0311 04:15:35.417276 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 344.175µs to process 0 records I0311 04:15:35.420708 1 tasks_processing.go:71] worker 26 working on nodenetworkstates task. I0311 04:15:35.420721 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 4.359219ms to process 0 records I0311 04:15:35.426683 1 tasks_processing.go:71] worker 24 working on infrastructures task. I0311 04:15:35.426700 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 9.917493ms to process 0 records I0311 04:15:35.426715 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 9.917312ms to process 0 records I0311 04:15:35.426724 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 9.978176ms to process 0 records I0311 04:15:35.426763 1 tasks_processing.go:71] worker 8 working on operators task. I0311 04:15:35.426763 1 tasks_processing.go:71] worker 16 working on storage_classes task. I0311 04:15:35.426924 1 tasks_processing.go:71] worker 7 working on overlapping_namespace_uids task. I0311 04:15:35.426934 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 9.406404ms to process 0 records I0311 04:15:35.427311 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 9.321955ms to process 0 records I0311 04:15:35.427320 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 10.755563ms to process 0 records I0311 04:15:35.427331 1 tasks_processing.go:71] worker 1 working on openstack_version task. I0311 04:15:35.427499 1 tasks_processing.go:71] worker 14 working on config_maps task. I0311 04:15:35.427500 1 tasks_processing.go:71] worker 10 working on mutating_webhook_configurations task. I0311 04:15:35.427518 1 controller.go:119] Initializing last reported time to 0001-01-01T00:00:00Z I0311 04:15:35.427536 1 controller.go:203] Source periodic-conditional *controllerstatus.Simple is not ready I0311 04:15:35.427543 1 controller.go:203] Source periodic-workloads *controllerstatus.Simple is not ready I0311 04:15:35.427548 1 controller.go:203] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0311 04:15:35.427563 1 controller.go:458] The operator is still being initialized I0311 04:15:35.427662 1 controller.go:481] The operator is healthy I0311 04:15:35.428244 1 recorder.go:70] Recording config/apiserver with fingerprint=65e072b285040261cf969d3b5046f9aab0241a9506920665936fa32a2611aa4c I0311 04:15:35.428260 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 11.038761ms to process 1 records I0311 04:15:35.428274 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 9.588251ms to process 0 records I0311 04:15:35.428279 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 10.777876ms to process 0 records I0311 04:15:35.428287 1 tasks_processing.go:71] worker 18 working on sap_datahubs task. I0311 04:15:35.428357 1 recorder.go:70] Recording config/schedulers/cluster with fingerprint=27c659b111fc3154fd1869b20d865493b14a8f12835dcc151ce070b6588fd2ca I0311 04:15:35.428363 1 tasks_processing.go:71] worker 2 working on sap_config task. I0311 04:15:35.428373 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 11.630927ms to process 1 records I0311 04:15:35.428374 1 tasks_processing.go:71] worker 13 working on olm_operators task. I0311 04:15:35.428671 1 tasks_processing.go:71] worker 0 working on pod_network_connectivity_checks task. I0311 04:15:35.430743 1 recorder.go:70] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=5ec1031560d43261b92b54cd00a209b1f67d662dd0332c3953b8090a23c0d373 I0311 04:15:35.430768 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 12.42344ms to process 1 records I0311 04:15:35.430780 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 12.892997ms to process 0 records I0311 04:15:35.430805 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 11.136622ms to process 0 records I0311 04:15:35.430817 1 tasks_processing.go:71] worker 9 working on networks task. I0311 04:15:35.430987 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 3.622194ms to process 0 records I0311 04:15:35.431259 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 13.007957ms to process 0 records E0311 04:15:35.431300 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0311 04:15:35.431267 1 tasks_processing.go:71] worker 19 working on container_runtime_configs task. I0311 04:15:35.431336 1 recorder.go:70] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0311 04:15:35.431349 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 14.799163ms to process 1 records I0311 04:15:35.431356 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 10.535707ms to process 0 records I0311 04:15:35.431364 1 tasks_processing.go:71] worker 26 working on dvo_metrics task. I0311 04:15:35.431388 1 tasks_processing.go:71] worker 20 working on certificate_signing_requests task. I0311 04:15:35.431443 1 tasks_processing.go:71] worker 21 working on machines task. I0311 04:15:35.431271 1 tasks_processing.go:71] worker 1 working on storage_cluster task. I0311 04:15:35.431285 1 tasks_processing.go:71] worker 29 working on install_plans task. I0311 04:15:35.431782 1 recorder.go:70] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=994827474e1c2cca25753714ddc36b0921a45ab1ceb3caea56073bb1c4151921 I0311 04:15:35.431912 1 recorder.go:70] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=e46c9e67314c98293002e93e6be9f6d38d1561ea34700627fde36804d83ddc27 I0311 04:15:35.431950 1 recorder.go:70] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=354672f8f58023acc00ac0f4755dcf9f3331d5f23b5e8de2a117ba1df107c193 I0311 04:15:35.431987 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=193552e7d4595880e273a425f4f3c42cc8d14f44c8035e6bdae701770fff6455 I0311 04:15:35.432019 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=f356dea26cfa91eecbe4e6571af111b8af880f12338b7671acade5e050ffe0f7 I0311 04:15:35.432052 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=e0f50a8eca62655a9e08801ea5c5ada41547d0938fe4104c972c0b0e6424e932 I0311 04:15:35.432092 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=5d6fc033201ae4f48d5958c656800547047e9b1543811eea8bfc6f0bb02eaa85 I0311 04:15:35.432125 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=7d087fddc5806baf4004b1a0a23feadaaec4eef634ed0559b3cd508e96c6958d I0311 04:15:35.432156 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=ba5ac64a2f58c004c3d64cfb58624fcf1bdb4c999f20b3976992ae639c54123f I0311 04:15:35.432186 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=be62304bcd5f01268d539c613f92c92d1caeae39059803d6a0d00a5cf5c37392 I0311 04:15:35.432196 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 15.042293ms to process 10 records I0311 04:15:35.433048 1 tasks_processing.go:71] worker 12 working on machine_healthchecks task. I0311 04:15:35.433324 1 recorder.go:70] Recording config/authentication with fingerprint=10ef0394bbe43f4550880931d3c593227e26d9031273b3841c564159987334f0 I0311 04:15:35.433339 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 16.078136ms to process 1 records I0311 04:15:35.433448 1 recorder.go:70] Recording config/featuregate with fingerprint=4b6b05725469e19b9a14d35c684fbb48608f28a6cdbc05b0e3affd992ef3beda I0311 04:15:35.433463 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 16.122676ms to process 1 records I0311 04:15:35.433448 1 tasks_processing.go:71] worker 4 working on ingress task. I0311 04:15:35.433567 1 recorder.go:70] Recording config/image with fingerprint=0c309447fa5de994a41bb6c437220257a0ebdff82ee44e8cc418ed0572c7b543 I0311 04:15:35.433581 1 gather.go:177] gatherer "clusterconfig" function "image" took 16.954124ms to process 1 records I0311 04:15:35.433590 1 tasks_processing.go:71] worker 11 working on crds task. I0311 04:15:35.433858 1 tasks_processing.go:71] worker 23 working on image_pruners task. I0311 04:15:35.433961 1 recorder.go:70] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=cee51cd64d4d8d0678179494ec89cfea0440f80656e349575fa770734c4cd978 I0311 04:15:35.434000 1 recorder.go:70] Recording config/pdbs/openshift-ingress/router-default with fingerprint=8b0c2d3ccb73ae9d52bfd566fb3bdc2253eec06ff660c9eae6478b5cf2e82363 I0311 04:15:35.434018 1 recorder.go:70] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=6161d94f8509c119e0376bc8f73dd767385209e4f5300fd308b522667bf65f77 I0311 04:15:35.434028 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 15.959871ms to process 3 records I0311 04:15:35.434276 1 tasks_processing.go:71] worker 17 working on proxies task. E0311 04:15:35.434284 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0311 04:15:35.434293 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 16.179903ms to process 0 records I0311 04:15:35.435101 1 tasks_processing.go:71] worker 24 working on machine_autoscalers task. I0311 04:15:35.435778 1 recorder.go:70] Recording config/infrastructure with fingerprint=7aee249c51940b2f7a1b053249d7b494b8f8970ff11a660ad5a81e9f3087b525 I0311 04:15:35.435793 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 8.400175ms to process 1 records I0311 04:15:35.435807 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 17.754396ms to process 0 records I0311 04:15:35.435820 1 tasks_processing.go:71] worker 5 working on cost_management_metrics_configs task. I0311 04:15:35.442817 1 tasks_processing.go:71] worker 16 working on metrics task. W0311 04:15:35.442845 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0311 04:15:35.442929 1 recorder.go:70] Recording config/storage/storageclasses/gp2-csi with fingerprint=a13f83ce634f2492b90ee9c18144146089e604d54823d3cb4b6859e44abf1bc0 I0311 04:15:35.442950 1 recorder.go:70] Recording config/storage/storageclasses/gp3-csi with fingerprint=4b97c44cb196d8ca8ba35b89431487e5c4250892c0728366774c55dceebe1ee7 I0311 04:15:35.442960 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 15.951198ms to process 2 records I0311 04:15:35.442971 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 24.365µs to process 0 records I0311 04:15:35.442977 1 tasks_processing.go:71] worker 16 working on jaegers task. I0311 04:15:35.444349 1 tasks_processing.go:71] worker 0 working on oauths task. E0311 04:15:35.444384 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0311 04:15:35.444544 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 15.504272ms to process 0 records I0311 04:15:35.444555 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 15.978795ms to process 0 records I0311 04:15:35.444563 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 13.041683ms to process 0 records I0311 04:15:35.444569 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 12.858721ms to process 0 records E0311 04:15:35.444580 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0311 04:15:35.444589 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 11.365334ms to process 0 records E0311 04:15:35.444596 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0311 04:15:35.444608 1 tasks_processing.go:74] worker 19 stopped. I0311 04:15:35.444619 1 tasks_processing.go:74] worker 2 stopped. I0311 04:15:35.444634 1 gather.go:177] gatherer "clusterconfig" function "machines" took 12.990986ms to process 0 records I0311 04:15:35.444640 1 tasks_processing.go:74] worker 1 stopped. I0311 04:15:35.444662 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 16.220995ms to process 0 records I0311 04:15:35.444670 1 tasks_processing.go:74] worker 12 stopped. I0311 04:15:35.444680 1 tasks_processing.go:74] worker 21 stopped. I0311 04:15:35.444686 1 tasks_processing.go:74] worker 18 stopped. I0311 04:15:35.444756 1 tasks_processing.go:74] worker 3 stopped. I0311 04:15:35.445068 1 recorder.go:70] Recording config/node/ip-10-0-0-72.ec2.internal with fingerprint=49f582b8f6fb6d8b9613dbc7c00b075de176151653148255c5c0a8522a933934 I0311 04:15:35.445471 1 recorder.go:70] Recording config/node/ip-10-0-1-71.ec2.internal with fingerprint=ceadd74c23df706014f6fed010d6a869211fe7d47c381c96303f6dae4ef29f8d I0311 04:15:35.445758 1 recorder.go:70] Recording config/node/ip-10-0-2-202.ec2.internal with fingerprint=e426e69e96f919c25aecc669938d21228dc332a164238c927ea2703516c8fd9f I0311 04:15:35.445847 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 26.837463ms to process 3 records I0311 04:15:35.445902 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 9.504603ms to process 0 records I0311 04:15:35.445942 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 8.80752ms to process 0 records I0311 04:15:35.446008 1 recorder.go:70] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0311 04:15:35.446071 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 18.508827ms to process 1 records I0311 04:15:35.446112 1 tasks_processing.go:74] worker 7 stopped. I0311 04:15:35.446138 1 tasks_processing.go:74] worker 24 stopped. I0311 04:15:35.446172 1 tasks_processing.go:74] worker 5 stopped. I0311 04:15:35.448140 1 tasks_processing.go:74] worker 16 stopped. I0311 04:15:35.448159 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 5.155911ms to process 0 records I0311 04:15:35.451145 1 tasks_processing.go:74] worker 13 stopped. I0311 04:15:35.451162 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 22.750925ms to process 0 records I0311 04:15:35.451229 1 tasks_processing.go:74] worker 9 stopped. I0311 04:15:35.451423 1 recorder.go:70] Recording config/network with fingerprint=7feb98c5e1c3962570ba4ba1d924d501847b052089dc383cbadf2facfb5ba73f I0311 04:15:35.451442 1 gather.go:177] gatherer "clusterconfig" function "networks" took 20.40428ms to process 1 records I0311 04:15:35.454067 1 tasks_processing.go:74] worker 0 stopped. I0311 04:15:35.454409 1 recorder.go:70] Recording config/oauth with fingerprint=46794a208e77cffdda7fdafd55fa8c9bce92e66b9ce61cb41f7d1c488a510af8 I0311 04:15:35.454430 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 9.662488ms to process 1 records I0311 04:15:35.454523 1 tasks_processing.go:74] worker 23 stopped. I0311 04:15:35.454547 1 recorder.go:70] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=d4e57a747814b1f0ad6dbe4b36e329fe481397706cd8516ff5d3b955242d6eb4 I0311 04:15:35.454557 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 20.241255ms to process 1 records I0311 04:15:35.454620 1 recorder.go:70] Recording config/proxy with fingerprint=35475757f098c979dadc5f67dc9ca5d6f0c8e97a80ee8fceee7a310b7ae1e8a0 I0311 04:15:35.454632 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 19.977823ms to process 1 records I0311 04:15:35.454666 1 tasks_processing.go:74] worker 17 stopped. I0311 04:15:35.454756 1 tasks_processing.go:74] worker 10 stopped. I0311 04:15:35.454857 1 recorder.go:70] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=fc8769d9cced9abf248d2e8eaad82faea742c4835758d4b72fe89e11b596b91c I0311 04:15:35.454897 1 recorder.go:70] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=6f8aae4391a004138c268e47825e0d5c179a5b93fdf008a40d5706107a9b0c34 I0311 04:15:35.454945 1 recorder.go:70] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=34f724c7d7ead0a42f8bea9e75f6bc74a43a548473495f98ba851bd973862573 I0311 04:15:35.454956 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 27.234154ms to process 3 records I0311 04:15:35.455168 1 tasks_processing.go:74] worker 4 stopped. I0311 04:15:35.455300 1 recorder.go:70] Recording config/ingress with fingerprint=00a2d7f5eaa09337f9ae83a748f0a5dbd1d15d225b8cb7752f2a5705aca0d9a6 I0311 04:15:35.455316 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 21.638698ms to process 1 records I0311 04:15:35.456233 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0311 04:15:35.456346 1 operator.go:287] started I0311 04:15:35.456380 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0311 04:15:35.456243 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0311 04:15:35.456919 1 tasks_processing.go:74] worker 28 stopped. I0311 04:15:35.457283 1 recorder.go:70] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=537e4cb5c186f41d77a13fab76091bf0595c3235e0cfd66b441b543831d48d89 I0311 04:15:35.457462 1 recorder.go:70] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=632a60723886dca72d234373c164c08e4bfb84d245605ecd5c53bc0dd0d1c982 I0311 04:15:35.457499 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 39.602253ms to process 2 records I0311 04:15:35.461584 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 43.095199ms to process 0 records I0311 04:15:35.461020 1 tasks_processing.go:74] worker 31 stopped. I0311 04:15:35.462215 1 tasks_processing.go:74] worker 20 stopped. I0311 04:15:35.462250 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 30.805708ms to process 0 records I0311 04:15:35.467937 1 tasks_processing.go:74] worker 6 stopped. I0311 04:15:35.468051 1 recorder.go:70] Recording config/running_containers with fingerprint=27243770c1d9be5edb16a577c414ad8dfb169707eb71f7faacc6edf2451cf803 I0311 04:15:35.468066 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 50.519701ms to process 1 records W0311 04:15:35.468946 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0311 04:15:35.472027 1 tasks_processing.go:74] worker 22 stopped. I0311 04:15:35.472042 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 54.793327ms to process 0 records I0311 04:15:35.475135 1 controller.go:203] Source periodic-workloads *controllerstatus.Simple is not ready I0311 04:15:35.475154 1 controller.go:203] Source scaController *sca.Controller is not ready I0311 04:15:35.475160 1 controller.go:203] Source clusterTransferController *clustertransfer.Controller is not ready I0311 04:15:35.475165 1 controller.go:203] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0311 04:15:35.475169 1 controller.go:203] Source periodic-conditional *controllerstatus.Simple is not ready I0311 04:15:35.475190 1 controller.go:458] The operator is still being initialized I0311 04:15:35.475202 1 controller.go:481] The operator is healthy E0311 04:15:35.478329 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%275412bef2-72f1-4d8d-bba2-fa15d7a7969a%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.13:43423->172.30.0.10:53: read: connection refused I0311 04:15:35.478343 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%275412bef2-72f1-4d8d-bba2-fa15d7a7969a%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.13:43423->172.30.0.10:53: read: connection refused I0311 04:15:35.481061 1 prometheus_rules.go:88] Prometheus rules successfully created I0311 04:15:35.481987 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0311 04:15:35.482001 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0311 04:15:35.481972 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0311 04:15:35.482402 1 tasks_processing.go:74] worker 11 stopped. I0311 04:15:35.483317 1 recorder.go:70] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=81a057e2d0fd6b7815f88a26522bf2de7d8f026e618fec7fafac5ef5dc855361 I0311 04:15:35.483567 1 recorder.go:70] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=48f359d307232c4c2a0f470a8d0f19cb22e8b7a7943a6b2b16d7db061527fbca I0311 04:15:35.483579 1 gather.go:177] gatherer "clusterconfig" function "crds" took 48.792258ms to process 2 records I0311 04:15:35.486448 1 base_controller.go:82] Caches are synced for ConfigController I0311 04:15:35.486462 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0311 04:15:35.492115 1 tasks_processing.go:74] worker 30 stopped. I0311 04:15:35.492624 1 recorder.go:70] Recording config/version with fingerprint=3ab1ed70bb1c634a40f57e9116e901828e485e6483e1b9902c8aab3da81a9598 I0311 04:15:35.492675 1 recorder.go:70] Recording config/id with fingerprint=49c99a422f2ee4ceb90479233b92fe92e2bbbd5f4c784a636451b748bd09122f I0311 04:15:35.492684 1 gather.go:177] gatherer "clusterconfig" function "version" took 75.54946ms to process 2 records I0311 04:15:35.504433 1 configmapobserver.go:84] configmaps "insights-config" not found I0311 04:15:35.506592 1 tasks_processing.go:74] worker 14 stopped. E0311 04:15:35.506606 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0311 04:15:35.506612 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0311 04:15:35.506615 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0311 04:15:35.506625 1 recorder.go:70] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=295a2bc7810c628251f1f312c209b5eb4da888326706c635ac0c12aec0969e17 I0311 04:15:35.506676 1 recorder.go:70] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0311 04:15:35.506684 1 recorder.go:70] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0311 04:15:35.506689 1 recorder.go:70] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=fe7abfa7f7aea852d8bca6b7df9b4d5de32045254ff2774f9bd30f0b6dcb7dc4 I0311 04:15:35.506693 1 recorder.go:70] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0311 04:15:35.506731 1 recorder.go:70] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0311 04:15:35.506739 1 recorder.go:70] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0311 04:15:35.506745 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 79.07141ms to process 7 records I0311 04:15:35.541636 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0311 04:15:35.551006 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.13:49960->172.30.0.10:53: read: connection refused I0311 04:15:35.551023 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.13:49960->172.30.0.10:53: read: connection refused I0311 04:15:35.556795 1 base_controller.go:82] Caches are synced for LoggingSyncer I0311 04:15:35.556814 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0311 04:15:35.561708 1 tasks_processing.go:74] worker 27 stopped. E0311 04:15:35.561725 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0311 04:15:35.561731 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2ov9k0fb8ou8ssp2k5rs588vm01vl8sk-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2ov9k0fb8ou8ssp2k5rs588vm01vl8sk-primary-cert-bundle-secret" not found I0311 04:15:35.561805 1 recorder.go:70] Recording aggregated/ingress_controllers_certs with fingerprint=8dc471ed0733ffa86980ddd069c9c6b19f92e168399eec4eeb09571c8f2b3aa6 I0311 04:15:35.561819 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 143.821998ms to process 1 records I0311 04:15:35.869844 1 gather_cluster_operator_pods_and_events.go:121] Found 16 pods with 18 containers I0311 04:15:35.869860 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1398101 bytes I0311 04:15:35.869894 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-722rp pod in namespace openshift-dns (previous: false). I0311 04:15:35.876585 1 gather_cluster_operators.go:184] Unable to get operatorpkis.network.operator.openshift.io resource due to: operatorpkis.network.operator.openshift.io "ovn" is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot get resource "operatorpkis" in API group "network.operator.openshift.io" in the namespace "openshift-ovn-kubernetes" I0311 04:15:36.075182 1 gather_cluster_operators.go:184] Unable to get operatorpkis.network.operator.openshift.io resource due to: operatorpkis.network.operator.openshift.io "signer" is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot get resource "operatorpkis" in API group "network.operator.openshift.io" in the namespace "openshift-ovn-kubernetes" I0311 04:15:36.109668 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-722rp pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-722rp\" is waiting to start: ContainerCreating" I0311 04:15:36.109687 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-722rp\" is waiting to start: ContainerCreating" I0311 04:15:36.109694 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-722rp pod in namespace openshift-dns (previous: false). I0311 04:15:36.266774 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-722rp pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-722rp\" is waiting to start: ContainerCreating" I0311 04:15:36.266792 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-722rp\" is waiting to start: ContainerCreating" I0311 04:15:36.266802 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-jzgvr pod in namespace openshift-dns (previous: false). W0311 04:15:36.463891 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0311 04:15:36.488635 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-jzgvr pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-jzgvr\" is waiting to start: ContainerCreating" I0311 04:15:36.488667 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-jzgvr\" is waiting to start: ContainerCreating" I0311 04:15:36.488675 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-jzgvr pod in namespace openshift-dns (previous: false). I0311 04:15:36.667115 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-jzgvr pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-jzgvr\" is waiting to start: ContainerCreating" I0311 04:15:36.667133 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-jzgvr\" is waiting to start: ContainerCreating" I0311 04:15:36.667144 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-kcbwz pod in namespace openshift-dns (previous: false). I0311 04:15:36.876811 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0311 04:15:36.894870 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for node-resolver-kcbwz pod in namespace openshift-dns for failing operator dns-node-resolver (previous: false): "container \"dns-node-resolver\" in pod \"node-resolver-kcbwz\" is waiting to start: ContainerCreating" I0311 04:15:36.894887 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns-node-resolver\" in pod \"node-resolver-kcbwz\" is waiting to start: ContainerCreating" I0311 04:15:36.894899 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-lxgdb pod in namespace openshift-dns (previous: false). I0311 04:15:37.065201 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0311 04:15:37.065219 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-vpj6z pod in namespace openshift-dns (previous: false). I0311 04:15:37.264967 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0311 04:15:37.264985 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-65d57fc87f-mr5ck pod in namespace openshift-image-registry (previous: false). I0311 04:15:37.279031 1 tasks_processing.go:74] worker 8 stopped. I0311 04:15:37.279083 1 recorder.go:70] Recording config/clusteroperator/console with fingerprint=49baa9e1fb033e8505d889abd45e22ebbe6714aa914160df26a3f0632440182c I0311 04:15:37.279113 1 recorder.go:70] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=58e9cf635ff2f64ed821b76d2f8ba6c9d8979f74f8fda54ec694ac6f5bba4905 I0311 04:15:37.279145 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0311 04:15:37.279182 1 recorder.go:70] Recording config/clusteroperator/dns with fingerprint=4a74ff3d7a4f74ea9fc924329799bb4c3cb85d713ee97b85c30fea4b836dcf7b I0311 04:15:37.279365 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0311 04:15:37.279422 1 recorder.go:70] Recording config/clusteroperator/image-registry with fingerprint=a6fbee4244db47fc1ce69a41ab3460d9381448872681c193e2d555686e678df3 I0311 04:15:37.279564 1 recorder.go:70] Recording config/clusteroperator/ingress with fingerprint=f6b32f650eb52676e631276e5adbbc70675cf61306473c775b682d9a077465e9 I0311 04:15:37.279676 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=d2c19ce04539768bbd6fe813d053d60f582e8f2d49c418ff7f1102b96703c7e4 I0311 04:15:37.279707 1 recorder.go:70] Recording config/clusteroperator/insights with fingerprint=c9fa5dc2d1b09063ce98eba60bdb0538db64cd7faa4dda1af62c8ca5390acbbb I0311 04:15:37.279749 1 recorder.go:70] Recording config/clusteroperator/kube-apiserver with fingerprint=aaa9a554e04dac9d89407a476873bf492c09bed9e2121039347df2231f7105fd I0311 04:15:37.279766 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0311 04:15:37.279793 1 recorder.go:70] Recording config/clusteroperator/kube-controller-manager with fingerprint=805c0a863c392f968887c470acd91a37c56f0da898deeaef5ff4d78fd0b1f305 I0311 04:15:37.279818 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0311 04:15:37.279848 1 recorder.go:70] Recording config/clusteroperator/kube-scheduler with fingerprint=d833a72278670a4b2972189428adf2a3c59925923367641e124b991efa4fbabb I0311 04:15:37.279865 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0311 04:15:37.279894 1 recorder.go:70] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=c7f968b3d1b7bf3ee8e15d258e3dba5af8e2e4cf1a156ab4c7b6dc34db6739a2 I0311 04:15:37.279910 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0311 04:15:37.279938 1 recorder.go:70] Recording config/clusteroperator/monitoring with fingerprint=788776351d231ff508a0e45407712049a65a44c2bb46a3db5959703731f04727 I0311 04:15:37.280144 1 recorder.go:70] Recording config/clusteroperator/network with fingerprint=2658143284145d6cba8a09e72bc0b9bc0f053822d1ca39b9c116aace15e91426 I0311 04:15:37.280208 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0311 04:15:37.280249 1 recorder.go:70] Recording config/clusteroperator/node-tuning with fingerprint=244365fb15dc8b54f6351bf48a1b190d7da0ec11dba7ddc73c726271e7ca8ab5 I0311 04:15:37.280298 1 recorder.go:70] Recording config/clusteroperator/openshift-apiserver with fingerprint=963f117aec9cf65fe292900fd623a70db554148ad17dffa1dcd4d4c199c9a7be I0311 04:15:37.280317 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0311 04:15:37.280345 1 recorder.go:70] Recording config/clusteroperator/openshift-controller-manager with fingerprint=05284ad51e71d199ac20b9a2b1b4f12f4e9313ed2a26f781aaba2b30c45e14a5 I0311 04:15:37.280367 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0311 04:15:37.280483 1 recorder.go:70] Recording config/clusteroperator/openshift-samples with fingerprint=14b8e4a6e830a1ebb4aa9422061e93ce996cfde2790971bfba52240b4ce8fdf5 I0311 04:15:37.280511 1 recorder.go:70] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=dec216140bb877df64a270ed0dcff763159b61c8a9c1223ef922bc7b32da0919 I0311 04:15:37.280543 1 recorder.go:70] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=4879c5f07ad4761c0678626b424ec3c2ff9cd3bb81528e723a6844dc8801b09c I0311 04:15:37.280598 1 recorder.go:70] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=f48eb6e3bf6e329015f10167c4bc978bc679f57f69b9081021914c648564eea4 I0311 04:15:37.280623 1 recorder.go:70] Recording config/clusteroperator/service-ca with fingerprint=c2bc679dc95effae91ab6a1291cc09895426e4795659b49e49bd01066893567e I0311 04:15:37.280684 1 recorder.go:70] Recording config/clusteroperator/storage with fingerprint=74deefd0db4ee86d8e4084af5a729e66707fc99aed9410fee47b42ba288fcae6 I0311 04:15:37.280720 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0311 04:15:37.280737 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0311 04:15:37.280747 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.852244356s to process 33 records I0311 04:15:37.464965 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-65d57fc87f-mr5ck pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-65d57fc87f-mr5ck\" is waiting to start: ContainerCreating" I0311 04:15:37.464990 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-65d57fc87f-mr5ck\" is waiting to start: ContainerCreating" I0311 04:15:37.465004 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-65d57fc87f-rxmst pod in namespace openshift-image-registry (previous: false). W0311 04:15:37.465036 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0311 04:15:37.666307 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-65d57fc87f-rxmst pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-65d57fc87f-rxmst\" is waiting to start: ContainerCreating" I0311 04:15:37.666328 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-65d57fc87f-rxmst\" is waiting to start: ContainerCreating" I0311 04:15:37.666342 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-79658987dd-szxtf pod in namespace openshift-image-registry (previous: false). I0311 04:15:37.861007 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0311 04:15:37.861032 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-fb84f pod in namespace openshift-image-registry (previous: false). I0311 04:15:38.067279 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0311 04:15:38.067299 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-j9zsc pod in namespace openshift-image-registry (previous: false). I0311 04:15:38.270688 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0311 04:15:38.270710 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-v8jp6 pod in namespace openshift-image-registry (previous: false). I0311 04:15:38.465560 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0311 04:15:38.465579 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5687d94c79-btkbv pod in namespace openshift-ingress (previous: false). W0311 04:15:38.465929 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0311 04:15:38.667456 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5687d94c79-btkbv pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5687d94c79-btkbv\" is waiting to start: ContainerCreating" I0311 04:15:38.667474 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5687d94c79-btkbv\" is waiting to start: ContainerCreating" I0311 04:15:38.667485 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7ccb56cdd6-44hg4 pod in namespace openshift-ingress (previous: false). I0311 04:15:38.867290 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-7ccb56cdd6-44hg4 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-7ccb56cdd6-44hg4\" is waiting to start: ContainerCreating" I0311 04:15:38.867309 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-7ccb56cdd6-44hg4\" is waiting to start: ContainerCreating" I0311 04:15:38.867325 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7ccb56cdd6-7v6xr pod in namespace openshift-ingress (previous: false). I0311 04:15:39.065265 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-7ccb56cdd6-7v6xr pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-7ccb56cdd6-7v6xr\" is waiting to start: ContainerCreating" I0311 04:15:39.065284 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-7ccb56cdd6-7v6xr\" is waiting to start: ContainerCreating" I0311 04:15:39.065294 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-djnfn pod in namespace openshift-ingress-canary (previous: false). I0311 04:15:39.268772 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-djnfn pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-djnfn\" is waiting to start: ContainerCreating" I0311 04:15:39.268789 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-djnfn\" is waiting to start: ContainerCreating" I0311 04:15:39.268800 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-jwndq pod in namespace openshift-ingress-canary (previous: false). W0311 04:15:39.464830 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0311 04:15:39.465394 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-jwndq pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-jwndq\" is waiting to start: ContainerCreating" I0311 04:15:39.465411 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-jwndq\" is waiting to start: ContainerCreating" I0311 04:15:39.465435 1 tasks_processing.go:74] worker 15 stopped. I0311 04:15:39.465551 1 recorder.go:70] Recording events/openshift-dns with fingerprint=a0b6db261dd520887805cd0b3c5c646a2c8e94d88eb7677b3c67458d4f8b153b I0311 04:15:39.465669 1 recorder.go:70] Recording events/openshift-image-registry with fingerprint=a7f7a4468f603737f0b17d8e016d8cc412497e989783f839f79fda0a6e75b236 I0311 04:15:39.465702 1 recorder.go:70] Recording events/openshift-ingress-operator with fingerprint=0976d5e301bc70aa22da07d70d478cc111bde5143632cec657ddc1442ef9b9e5 I0311 04:15:39.465748 1 recorder.go:70] Recording events/openshift-ingress with fingerprint=75a508bc690db074ea26340a8fb68cc6ff0805fa2c945e7eb2b42e462ba85469 I0311 04:15:39.465763 1 recorder.go:70] Recording events/openshift-ingress-canary with fingerprint=31f168fa239528b2677d8aac58057f547661fb9b869552f09172ec8cac55420a I0311 04:15:39.465771 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 4.047474171s to process 5 records W0311 04:15:40.465406 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0311 04:15:40.465436 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0311 04:15:40.465448 1 tasks_processing.go:74] worker 26 stopped. E0311 04:15:40.465458 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0311 04:15:40.465470 1 recorder.go:70] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0311 04:15:40.465484 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0311 04:15:40.465496 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.034075409s to process 1 records I0311 04:15:47.741077 1 configmapobserver.go:84] configmaps "insights-config" not found I0311 04:15:47.867806 1 tasks_processing.go:74] worker 29 stopped. I0311 04:15:47.867850 1 recorder.go:70] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0311 04:15:47.867865 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.436086828s to process 1 records I0311 04:15:48.625620 1 tasks_processing.go:74] worker 25 stopped. I0311 04:15:48.626060 1 recorder.go:70] Recording config/serviceaccounts with fingerprint=6a25405fe5ae5697551c08cd61c8ee577eba432556fcc5dc2605b343adedffa8 I0311 04:15:48.626087 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.208549256s to process 1 records E0311 04:15:48.626167 1 periodic.go:250] "Unhandled Error" err="clusterconfig failed after 13.21s with: function \"machine_configs\" failed with an error, function \"support_secret\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0311 04:15:48.627273 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machine_configs" failed with an error, function "support_secret" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0311 04:15:48.627294 1 periodic.go:212] Running workloads gatherer I0311 04:15:48.627312 1 tasks_processing.go:45] number of workers: 2 I0311 04:15:48.627321 1 tasks_processing.go:69] worker 1 listening for tasks. I0311 04:15:48.627327 1 tasks_processing.go:71] worker 1 working on helmchart_info task. I0311 04:15:48.627335 1 tasks_processing.go:69] worker 0 listening for tasks. I0311 04:15:48.627412 1 tasks_processing.go:71] worker 0 working on workload_info task. I0311 04:15:48.653854 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 21s for image data I0311 04:15:48.670560 1 gather_workloads_info.go:387] No image sha256:a7c71f3c9714cf1717c45e12ec817be6d6209f00989e6a31d634e527f0c8147d (17ms) I0311 04:15:48.675584 1 tasks_processing.go:74] worker 1 stopped. I0311 04:15:48.675607 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 48.242285ms to process 0 records I0311 04:15:48.681903 1 gather_workloads_info.go:387] No image sha256:bae7a33f8db8a3d4b3c4c05498aba85a0ce463f85322067c48e79663710e616e (11ms) I0311 04:15:48.694296 1 gather_workloads_info.go:387] No image sha256:ccdda82b4adfa8ee4a84b3bad19693be6a42a9881a7b83788d835912129c49a4 (12ms) I0311 04:15:48.706284 1 gather_workloads_info.go:387] No image sha256:f2c46054a8f64a0e949cbf30295f31b4a35a0203c6ad03fa7ec922b4101dcbf8 (12ms) I0311 04:15:48.721748 1 gather_workloads_info.go:387] No image sha256:7941b5d9a758b8667e99cd9236f7f96ec036af61df5cbc96e16077d36700d7c7 (15ms) I0311 04:15:48.735904 1 gather_workloads_info.go:387] No image sha256:824a3b8f78e19aa21a9f6444aefe8e0b624886cca18fe828b13acbab55e6e868 (14ms) I0311 04:15:48.751552 1 gather_workloads_info.go:387] No image sha256:d28734effaadd66434e77a1a5fbe2e8a4ca2066cd9f8868c22ade9475539bfd7 (16ms) I0311 04:15:48.762605 1 gather_workloads_info.go:387] No image sha256:f6e156f39b06d196b46d8ed3a03ca337e939dba0060e40de4455b4e77b90929e (11ms) I0311 04:15:48.778523 1 gather_workloads_info.go:387] No image sha256:6d6ebff54b8adac74f4d1b12ac8aa0f16cd7b28370e0d6aa847d1c457e03a5b6 (16ms) I0311 04:15:48.791889 1 gather_workloads_info.go:387] No image sha256:87efe06afd1f04426fca7f86c0f74c4ee75c311ba199ddabd5b849b877bc59fa (13ms) I0311 04:15:48.804852 1 gather_workloads_info.go:387] No image sha256:23c4fe84047a4e6cbe7f75f470e7d6fe0e61e7910dd17945b1b61bc4b72f3f2a (13ms) I0311 04:15:48.872967 1 gather_workloads_info.go:387] No image sha256:8b1e32ce43eb1247b0558230b6e0baa85ac62f1a5e2a2089d40b4fea90538529 (68ms) I0311 04:15:48.964388 1 gather_workloads_info.go:387] No image sha256:18956d5fbc8d0369a26cb1e00eb5c3aff1973ae8cbd3983920a0cf2851f05358 (91ms) I0311 04:15:49.068060 1 gather_workloads_info.go:387] No image sha256:0a8a0473029ace3adbb66f490dbe560f0e7782a38cdf32c1f7dd3e092e1d191e (104ms) I0311 04:15:49.164303 1 gather_workloads_info.go:387] No image sha256:373614619b9420b110d1508c5f17e066ce69c4c226fd04d02fbb959d9ba41eb6 (96ms) I0311 04:15:49.268154 1 gather_workloads_info.go:387] No image sha256:0a92dd43975f972e3dc707fb37854e046813531553038147e9c90d54a8d9df73 (104ms) I0311 04:15:49.365778 1 gather_workloads_info.go:387] No image sha256:3d0634fe58641d5242649c44ddba70ed67fdd6d2dcb5c2261df5cee8b33de9fd (98ms) I0311 04:15:49.467770 1 gather_workloads_info.go:387] No image sha256:9697bc2258bdfa9ae8c1866cc7eb0b3b46851998db827b121bdf77417a881eb3 (102ms) I0311 04:15:49.566915 1 gather_workloads_info.go:387] No image sha256:2d077e73dd76873ee2a1583aa3b3e76a1b408737f0af2e22b2aa055604d89e81 (99ms) I0311 04:15:49.667761 1 gather_workloads_info.go:387] No image sha256:8ef4bdf07dda423fab73484dbc66d527ce8ba8cc2d4f99210bc1bc24ea08c0cb (101ms) I0311 04:15:49.766054 1 gather_workloads_info.go:387] No image sha256:48404bd61d05dc738ccce4d22e36b30dbcdc6015b06b4afb604fd1baeee35bf2 (98ms) I0311 04:15:49.865827 1 gather_workloads_info.go:387] No image sha256:420c12f61c53e54eab2d99476759c0de339fca98ce0a3a782bc6545cc0e97a9c (100ms) I0311 04:15:49.966875 1 gather_workloads_info.go:387] No image sha256:752be69b2262be713df12c47f4bac8c2dafed272c401e9a89f8060f053d68054 (101ms) I0311 04:15:50.067902 1 gather_workloads_info.go:387] No image sha256:ceab8a8c340801fcebf800620d3fef5493d8a443077a21c6e56984051bd3abde (101ms) I0311 04:15:50.069217 1 configmapobserver.go:84] configmaps "insights-config" not found I0311 04:15:50.166006 1 gather_workloads_info.go:387] No image sha256:76f820bd9bd138d29305d545d7d49bfe63e2923f1fb1ec2e8eac81a388359024 (98ms) I0311 04:15:50.267835 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (102ms) I0311 04:15:50.270745 1 configmapobserver.go:84] configmaps "insights-config" not found I0311 04:15:50.367276 1 gather_workloads_info.go:387] No image sha256:944d9261ba7a143131fe8267c172defc5f37acc1cea3d4d373ec6fc5d8bfcc31 (99ms) I0311 04:15:50.467388 1 gather_workloads_info.go:387] No image sha256:4f175ee49f51dc4379e9993fecd1657c7a9c4c3dc096772cb198d0212b9eea47 (100ms) I0311 04:15:50.569710 1 gather_workloads_info.go:387] No image sha256:2fac754deaeade3456361eed52e344318ff16d04819384432759f0ea35530114 (102ms) I0311 04:15:50.667727 1 gather_workloads_info.go:387] No image sha256:6e67c980b7300e769fb4b2adaaf006d0f8274e43b10586701b204ea5153f15fc (98ms) I0311 04:15:50.766226 1 gather_workloads_info.go:387] No image sha256:14703f73bb1ca69ea03b726e340b3b68f8e294e1f22c80d1a16666ea3d4a88a3 (98ms) I0311 04:15:50.867472 1 gather_workloads_info.go:387] No image sha256:a9e885d6f0456a2cc10f9e5da71fe9403f5e4c639b9b7ea15bd403b272ccb824 (101ms) I0311 04:15:50.967631 1 gather_workloads_info.go:387] No image sha256:d93441b88bd020cae0a819977b23ca7be19e037d9bf634c16710864a08a4929f (100ms) I0311 04:15:50.967753 1 tasks_processing.go:74] worker 0 stopped. E0311 04:15:50.967776 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0311 04:15:50.968186 1 recorder.go:70] Recording config/workload_info with fingerprint=0df1180ee3ee45db3c0c2129a1798919ee142229b85f7c6677058d1da63d15a1 I0311 04:15:50.968249 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.340324073s to process 1 records E0311 04:15:50.968308 1 periodic.go:250] "Unhandled Error" err="workloads failed after 2.34s with: function \"workload_info\" failed with an error" I0311 04:15:50.969441 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0311 04:15:50.969501 1 periodic.go:212] Running conditional gatherer I0311 04:15:50.978973 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.19.9/gathering_rules I0311 04:15:50.989183 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.19.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.129.0.13:54580->172.30.0.10:53: read: connection refused E0311 04:15:50.989598 1 conditional_gatherer.go:320] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0311 04:15:50.989692 1 conditional_gatherer.go:382] updating version cache for conditional gatherer I0311 04:15:50.999168 1 conditional_gatherer.go:390] cluster version is '4.19.9' E0311 04:15:50.999230 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 04:15:50.999261 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 04:15:50.999287 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 04:15:50.999312 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 04:15:50.999336 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 04:15:50.999360 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 04:15:50.999384 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 04:15:50.999407 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 04:15:50.999429 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing I0311 04:15:50.999468 1 tasks_processing.go:45] number of workers: 3 I0311 04:15:50.999516 1 tasks_processing.go:69] worker 2 listening for tasks. I0311 04:15:50.999560 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0311 04:15:50.999600 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0311 04:15:50.999518 1 tasks_processing.go:69] worker 0 listening for tasks. I0311 04:15:50.999709 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0311 04:15:50.999528 1 tasks_processing.go:69] worker 1 listening for tasks. I0311 04:15:50.999788 1 tasks_processing.go:74] worker 1 stopped. I0311 04:15:50.999765 1 recorder.go:70] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0311 04:15:50.999862 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.752µs to process 1 records I0311 04:15:50.999953 1 tasks_processing.go:74] worker 0 stopped. I0311 04:15:50.999979 1 recorder.go:70] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0311 04:15:51.000015 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.334µs to process 1 records I0311 04:15:51.000053 1 tasks_processing.go:74] worker 2 stopped. I0311 04:15:51.000072 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 425.338µs to process 0 records I0311 04:15:51.000117 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.19.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.129.0.13:54580->172.30.0.10:53: read: connection refused I0311 04:15:51.000144 1 recorder.go:70] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0311 04:15:51.026864 1 recorder.go:70] Recording insights-operator/gathers with fingerprint=015448d6d505431b1fcd700d0ccf3419a77aad53c73b6a7cea1ec7696555f69b I0311 04:15:51.027025 1 diskrecorder.go:70] Writing 95 records to /var/lib/insights-operator/insights-2026-03-11-041551.tar.gz I0311 04:15:51.035461 1 diskrecorder.go:51] Wrote 95 records to disk in 8ms I0311 04:15:51.035507 1 periodic.go:281] Gathering cluster info every 2h0m0s I0311 04:15:51.035531 1 periodic.go:282] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0311 04:16:03.052461 1 configmapobserver.go:84] configmaps "insights-config" not found I0311 04:16:49.963299 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="7880c01cbe17b3f511d5098baa256566f72eb7d2f2d7a95d7b05f74e645824a3") W0311 04:16:49.963336 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0311 04:16:49.963380 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="24631360c44a3013ebc3294c1ea431a09b11fbe03c1cf9801534c63642cb73d5") I0311 04:16:49.963388 1 periodic.go:173] Shutting down I0311 04:16:49.963401 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0311 04:16:49.963438 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0311 04:16:49.963433 1 base_controller.go:181] Shutting down LoggingSyncer ... I0311 04:16:49.963452 1 base_controller.go:181] Shutting down ConfigController ... I0311 04:16:49.963452 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0311 04:16:49.963477 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="2767de78b35d6d6d7d3f3896f905b140dffe9e54686a2bf8e95f0d896b54bc21") I0311 04:16:49.963479 1 requestheader_controller.go:194] Shutting down RequestHeaderAuthRequestController I0311 04:16:49.963492 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0311 04:16:49.963504 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0311 04:16:49.963509 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0311 04:16:49.963515 1 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" I0311 04:16:49.963548 1 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/tmp/serving-cert-3822289155/tls.crt::/tmp/serving-cert-3822289155/tls.key" I0311 04:16:49.963558 1 secure_serving.go:258] Stopped listening on [::]:8443 I0311 04:16:49.963576 1 genericapiserver.go:600] "[graceful-termination] shutdown event" name="HTTPServerStoppedListening" I0311 04:16:49.963589 1 base_controller.go:123] Shutting down worker of ConfigController controller ... E0311 04:16:49.963594 1 controller.go:290] Unable to write cluster operator status: client rate limiter Wait returned an error: context canceled I0311 04:16:49.963596 1 base_controller.go:113] All ConfigController workers have been terminated I0311 04:16:49.963604 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0311 04:16:49.963615 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0311 04:16:49.963615 1 genericapiserver.go:651] "[graceful-termination] not going to wait for active watch request(s) to drain"