I0311 06:25:56.936617 1 cmd.go:253] Using service-serving-cert provided certificates I0311 06:25:56.937093 1 observer_polling.go:159] Starting file observer W0311 06:25:56.956468 1 builder.go:272] unable to get owner reference (falling back to namespace): replicasets.apps "insights-operator-5ff5cb4f99" is forbidden: User "system:serviceaccount:openshift-insights:operator" cannot get resource "replicasets" in API group "apps" in the namespace "openshift-insights" I0311 06:25:57.375983 1 operator.go:59] Starting insights-operator v0.0.0-master+$Format:%H$ I0311 06:25:57.376291 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0311 06:25:57.377577 1 secure_serving.go:57] Forcing use of http/1.1 only W0311 06:25:57.377682 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0311 06:25:57.377713 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0311 06:25:57.377748 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0311 06:25:57.377771 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0311 06:25:57.377794 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0311 06:25:57.377816 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0311 06:25:57.378110 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0311 06:25:57.381634 1 operator.go:124] FeatureGates initialized: knownFeatureGates=[AWSEFSDriverVolumeMetrics AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BareMetalLoadBalancer BuildCSIVolumes CPMSMachineNamePrefix ChunkSizeMiB CloudDualStackNodeIPs ConsolePluginContentSecurityPolicy DisableKubeletCloudCredentialProviders ExternalOIDC GCPLabelsTags GatewayAPI GatewayAPIController HardwareSpeed IngressControllerLBSubnetsAWS KMSv1 ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles MultiArchInstallAWS MultiArchInstallGCP NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation NodeDisruptionPolicy OnClusterBuild PersistentIPsForVirtualization PrivateHostedZoneAWS RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController VSphereDriverConfiguration VSphereMultiVCenters ValidatingAdmissionPolicy AWSClusterHostedDNS AutomatedEtcdBackup BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNS GCPCustomAPIEndpoints HighlyAvailableArbiter ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather InsightsRuntimeExtractor KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController MachineAPIProviderOpenStack MachineConfigNodes MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PinnedImages PlatformOperators ProcMountType RouteAdvertisements SELinuxChangePolicy SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerification SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass VolumeGroupSnapshot] I0311 06:25:57.381721 1 event.go:377] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"openshift-insights", Name:"openshift-insights", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AWSEFSDriverVolumeMetrics", "AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BareMetalLoadBalancer", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ChunkSizeMiB", "CloudDualStackNodeIPs", "ConsolePluginContentSecurityPolicy", "DisableKubeletCloudCredentialProviders", "ExternalOIDC", "GCPLabelsTags", "GatewayAPI", "GatewayAPIController", "HardwareSpeed", "IngressControllerLBSubnetsAWS", "KMSv1", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "MultiArchInstallAWS", "MultiArchInstallGCP", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "NodeDisruptionPolicy", "OnClusterBuild", "PersistentIPsForVirtualization", "PrivateHostedZoneAWS", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "VSphereDriverConfiguration", "VSphereMultiVCenters", "ValidatingAdmissionPolicy"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AutomatedEtcdBackup", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "HighlyAvailableArbiter", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "InsightsRuntimeExtractor", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "MachineAPIProviderOpenStack", "MachineConfigNodes", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PinnedImages", "PlatformOperators", "ProcMountType", "RouteAdvertisements", "SELinuxChangePolicy", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0311 06:25:57.383205 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0311 06:25:57.383222 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0311 06:25:57.383229 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0311 06:25:57.383235 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0311 06:25:57.383246 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0311 06:25:57.383251 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0311 06:25:57.383468 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0311 06:25:57.383790 1 secure_serving.go:213] Serving securely on [::]:8443 I0311 06:25:57.383831 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" E0311 06:25:57.384503 1 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:serviceaccount:openshift-insights:operator\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-insights\"" event="&Event{ObjectMeta:{openshift-insights.189bb56cd7c634bf openshift-insights 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Namespace,Namespace:openshift-insights,Name:openshift-insights,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:FeatureGatesInitialized,Message:FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{\"AWSEFSDriverVolumeMetrics\", \"AdditionalRoutingCapabilities\", \"AdminNetworkPolicy\", \"AlibabaPlatform\", \"AzureWorkloadIdentity\", \"BareMetalLoadBalancer\", \"BuildCSIVolumes\", \"CPMSMachineNamePrefix\", \"ChunkSizeMiB\", \"CloudDualStackNodeIPs\", \"ConsolePluginContentSecurityPolicy\", \"DisableKubeletCloudCredentialProviders\", \"ExternalOIDC\", \"GCPLabelsTags\", \"GatewayAPI\", \"GatewayAPIController\", \"HardwareSpeed\", \"IngressControllerLBSubnetsAWS\", \"KMSv1\", \"ManagedBootImages\", \"ManagedBootImagesAWS\", \"MetricsCollectionProfiles\", \"MultiArchInstallAWS\", \"MultiArchInstallGCP\", \"NetworkDiagnosticsConfig\", \"NetworkLiveMigration\", \"NetworkSegmentation\", \"NodeDisruptionPolicy\", \"OnClusterBuild\", \"PersistentIPsForVirtualization\", \"PrivateHostedZoneAWS\", \"RouteExternalCertificate\", \"ServiceAccountTokenNodeBinding\", \"SetEIPForNLBIngressController\", \"VSphereDriverConfiguration\", \"VSphereMultiVCenters\", \"ValidatingAdmissionPolicy\"}, Disabled:[]v1.FeatureGateName{\"AWSClusterHostedDNS\", \"AutomatedEtcdBackup\", \"BootcNodeManagement\", \"ClusterAPIInstall\", \"ClusterAPIInstallIBMCloud\", \"ClusterMonitoringConfig\", \"ClusterVersionOperatorConfiguration\", \"DNSNameResolver\", \"DualReplica\", \"DyanmicServiceEndpointIBMCloud\", \"DynamicResourceAllocation\", \"EtcdBackendQuota\", \"EventedPLEG\", \"Example\", \"Example2\", \"ExternalOIDCWithUIDAndExtraClaimMappings\", \"GCPClusterHostedDNS\", \"GCPCustomAPIEndpoints\", \"HighlyAvailableArbiter\", \"ImageStreamImportMode\", \"IngressControllerDynamicConfigurationManager\", \"InsightsConfig\", \"InsightsConfigAPI\", \"InsightsOnDemandDataGather\", \"InsightsRuntimeExtractor\", \"KMSEncryptionProvider\", \"MachineAPIMigration\", \"MachineAPIOperatorDisableMachineHealthCheckController\", \"MachineAPIProviderOpenStack\", \"MachineConfigNodes\", \"MaxUnavailableStatefulSet\", \"MinimumKubeletVersion\", \"MixedCPUsAllocation\", \"MultiArchInstallAzure\", \"NewOLM\", \"NewOLMCatalogdAPIV1Metas\", \"NewOLMOwnSingleNamespace\", \"NewOLMPreflightPermissionChecks\", \"NodeSwap\", \"NutanixMultiSubnets\", \"OVNObservability\", \"OpenShiftPodSecurityAdmission\", \"PinnedImages\", \"PlatformOperators\", \"ProcMountType\", \"RouteAdvertisements\", \"SELinuxChangePolicy\", \"SELinuxMount\", \"ShortCertRotation\", \"SignatureStores\", \"SigstoreImageVerification\", \"SigstoreImageVerificationPKI\", \"TranslateStreamCloseWebsocketRequests\", \"UpgradeStatus\", \"UserNamespacesPodSecurityStandards\", \"UserNamespacesSupport\", \"VSphereConfigurableMaxAllowedBlockVolumesPerNode\", \"VSphereHostVMGroupZonal\", \"VSphereMultiDisk\", \"VSphereMultiNetworks\", \"VolumeAttributesClass\", \"VolumeGroupSnapshot\"}},Source:EventSource{Component:openshift-insights-operator,Host:,},FirstTimestamp:2026-03-11 06:25:57.381608639 +0000 UTC m=+0.487621362,LastTimestamp:2026-03-11 06:25:57.381608639 +0000 UTC m=+0.487621362,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:openshift-insights-operator,ReportingInstance:,}" W0311 06:25:57.386542 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0311 06:25:57.386590 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0311 06:25:57.386725 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0311 06:25:57.391660 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0311 06:25:57.391677 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0311 06:25:57.395690 1 secretconfigobserver.go:119] support secret does not exist I0311 06:25:57.399786 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0311 06:25:57.403587 1 secretconfigobserver.go:119] support secret does not exist I0311 06:25:57.406307 1 recorder.go:156] Pruning old reports every 4h24m40s, max age is 288h0m0s I0311 06:25:57.410638 1 periodic.go:212] Running clusterconfig gatherer I0311 06:25:57.410652 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0311 06:25:57.410660 1 insightsreport.go:296] Starting report retriever I0311 06:25:57.410666 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0311 06:25:57.410686 1 tasks_processing.go:45] number of workers: 32 I0311 06:25:57.410727 1 tasks_processing.go:69] worker 31 listening for tasks. I0311 06:25:57.410732 1 tasks_processing.go:69] worker 14 listening for tasks. I0311 06:25:57.410744 1 tasks_processing.go:69] worker 22 listening for tasks. I0311 06:25:57.410746 1 tasks_processing.go:71] worker 14 working on node_logs task. I0311 06:25:57.410751 1 tasks_processing.go:69] worker 28 listening for tasks. I0311 06:25:57.410753 1 tasks_processing.go:69] worker 15 listening for tasks. I0311 06:25:57.410758 1 tasks_processing.go:69] worker 16 listening for tasks. I0311 06:25:57.410760 1 tasks_processing.go:69] worker 27 listening for tasks. I0311 06:25:57.410764 1 tasks_processing.go:69] worker 17 listening for tasks. I0311 06:25:57.410765 1 tasks_processing.go:69] worker 29 listening for tasks. I0311 06:25:57.410771 1 tasks_processing.go:69] worker 24 listening for tasks. I0311 06:25:57.410771 1 tasks_processing.go:69] worker 18 listening for tasks. I0311 06:25:57.410775 1 tasks_processing.go:69] worker 7 listening for tasks. I0311 06:25:57.410779 1 tasks_processing.go:69] worker 19 listening for tasks. I0311 06:25:57.410638 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0311 06:25:57.410786 1 tasks_processing.go:69] worker 25 listening for tasks. I0311 06:25:57.410783 1 tasks_processing.go:69] worker 30 listening for tasks. I0311 06:25:57.410789 1 tasks_processing.go:69] worker 4 listening for tasks. I0311 06:25:57.410794 1 tasks_processing.go:69] worker 26 listening for tasks. I0311 06:25:57.410791 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0311 06:25:57.410798 1 tasks_processing.go:69] worker 5 listening for tasks. I0311 06:25:57.410803 1 tasks_processing.go:69] worker 21 listening for tasks. I0311 06:25:57.410801 1 tasks_processing.go:69] worker 3 listening for tasks. I0311 06:25:57.410809 1 tasks_processing.go:69] worker 11 listening for tasks. I0311 06:25:57.410812 1 tasks_processing.go:69] worker 13 listening for tasks. I0311 06:25:57.410818 1 tasks_processing.go:69] worker 1 listening for tasks. I0311 06:25:57.410819 1 tasks_processing.go:69] worker 10 listening for tasks. I0311 06:25:57.410821 1 tasks_processing.go:69] worker 12 listening for tasks. I0311 06:25:57.410820 1 tasks_processing.go:69] worker 9 listening for tasks. I0311 06:25:57.410826 1 tasks_processing.go:69] worker 2 listening for tasks. I0311 06:25:57.410829 1 tasks_processing.go:69] worker 8 listening for tasks. I0311 06:25:57.410833 1 tasks_processing.go:71] worker 27 working on operators task. I0311 06:25:57.410836 1 tasks_processing.go:71] worker 2 working on version task. I0311 06:25:57.410838 1 tasks_processing.go:71] worker 17 working on nodenetworkstates task. I0311 06:25:57.410844 1 tasks_processing.go:71] worker 24 working on dvo_metrics task. I0311 06:25:57.410841 1 tasks_processing.go:69] worker 6 listening for tasks. I0311 06:25:57.410862 1 tasks_processing.go:71] worker 6 working on infrastructures task. I0311 06:25:57.410896 1 tasks_processing.go:71] worker 29 working on ingress_certificates task. I0311 06:25:57.411061 1 tasks_processing.go:71] worker 21 working on storage_classes task. I0311 06:25:57.411114 1 tasks_processing.go:71] worker 18 working on ceph_cluster task. I0311 06:25:57.411200 1 tasks_processing.go:71] worker 22 working on container_runtime_configs task. I0311 06:25:57.411309 1 tasks_processing.go:71] worker 15 working on support_secret task. I0311 06:25:57.411359 1 tasks_processing.go:71] worker 16 working on olm_operators task. I0311 06:25:57.411398 1 tasks_processing.go:71] worker 30 working on machine_autoscalers task. I0311 06:25:57.411416 1 tasks_processing.go:71] worker 1 working on container_images task. I0311 06:25:57.411449 1 tasks_processing.go:71] worker 12 working on pod_network_connectivity_checks task. I0311 06:25:57.411519 1 tasks_processing.go:71] worker 26 working on lokistack task. I0311 06:25:57.411541 1 tasks_processing.go:71] worker 5 working on sap_datahubs task. I0311 06:25:57.411590 1 tasks_processing.go:71] worker 9 working on pdbs task. I0311 06:25:57.411666 1 tasks_processing.go:71] worker 10 working on oauths task. I0311 06:25:57.411708 1 tasks_processing.go:71] worker 4 working on metrics task. I0311 06:25:57.411726 1 tasks_processing.go:71] worker 28 working on nodenetworkconfigurationpolicies task. I0311 06:25:57.411764 1 tasks_processing.go:71] worker 19 working on image task. I0311 06:25:57.411779 1 tasks_processing.go:71] worker 25 working on proxies task. I0311 06:25:57.411797 1 tasks_processing.go:71] worker 11 working on config_maps task. I0311 06:25:57.411823 1 tasks_processing.go:71] worker 3 working on cluster_apiserver task. I0311 06:25:57.411845 1 tasks_processing.go:71] worker 13 working on machine_configs task. I0311 06:25:57.410800 1 tasks_processing.go:69] worker 20 listening for tasks. I0311 06:25:57.411954 1 tasks_processing.go:71] worker 20 working on authentication task. I0311 06:25:57.410744 1 tasks_processing.go:69] worker 23 listening for tasks. I0311 06:25:57.410738 1 tasks_processing.go:71] worker 31 working on active_alerts task. I0311 06:25:57.410782 1 tasks_processing.go:69] worker 0 listening for tasks. I0311 06:25:57.410834 1 tasks_processing.go:71] worker 7 working on tsdb_status task. I0311 06:25:57.412103 1 tasks_processing.go:71] worker 0 working on install_plans task. I0311 06:25:57.412124 1 tasks_processing.go:71] worker 23 working on nodes task. I0311 06:25:57.410837 1 tasks_processing.go:71] worker 8 working on monitoring_persistent_volumes task. I0311 06:25:57.415694 1 tasks_processing.go:71] worker 17 working on jaegers task. I0311 06:25:57.415710 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 4.83952ms to process 0 records I0311 06:25:57.415960 1 tasks_processing.go:71] worker 28 working on overlapping_namespace_uids task. I0311 06:25:57.415975 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 4.218525ms to process 0 records E0311 06:25:57.416344 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.121.212:9091: connect: connection refused I0311 06:25:57.416367 1 tasks_processing.go:71] worker 7 working on sap_config task. E0311 06:25:57.416381 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.121.212:9091: connect: connection refused E0311 06:25:57.416385 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.121.212:9091: connect: connection refused I0311 06:25:57.416402 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 4.292775ms to process 0 records E0311 06:25:57.416415 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.121.212:9091: connect: connection refused I0311 06:25:57.416435 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 4.685705ms to process 0 records I0311 06:25:57.416454 1 tasks_processing.go:71] worker 4 working on validating_webhook_configurations task. I0311 06:25:57.418581 1 tasks_processing.go:71] worker 5 working on operators_pods_and_events task. I0311 06:25:57.418592 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 7.005931ms to process 0 records I0311 06:25:57.418654 1 tasks_processing.go:71] worker 12 working on feature_gates task. E0311 06:25:57.418665 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0311 06:25:57.418678 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 7.186622ms to process 0 records I0311 06:25:57.418694 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 7.406719ms to process 0 records I0311 06:25:57.418707 1 tasks_processing.go:71] worker 22 working on openstack_dataplanenodesets task. I0311 06:25:57.418773 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 7.239384ms to process 0 records I0311 06:25:57.418802 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 7.631408ms to process 0 records I0311 06:25:57.418824 1 tasks_processing.go:71] worker 18 working on machine_sets task. I0311 06:25:57.418841 1 tasks_processing.go:71] worker 26 working on openshift_machine_api_events task. I0311 06:25:57.419290 1 tasks_processing.go:71] worker 30 working on openstack_version task. I0311 06:25:57.419300 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 7.876475ms to process 0 records I0311 06:25:57.419606 1 tasks_processing.go:71] worker 6 working on image_registries task. I0311 06:25:57.420466 1 recorder.go:70] Recording config/infrastructure with fingerprint=a97a75240ab6d8561e3bd1b7843bde941120a6f8d525629a5dbbdcd589d830b6 I0311 06:25:57.420483 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 8.73214ms to process 1 records I0311 06:25:57.421471 1 tasks_processing.go:71] worker 13 working on silenced_alerts task. E0311 06:25:57.421511 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0311 06:25:57.421545 1 recorder.go:70] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0311 06:25:57.421578 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 9.609858ms to process 1 records I0311 06:25:57.421768 1 controller.go:119] Initializing last reported time to 0001-01-01T00:00:00Z I0311 06:25:57.421790 1 controller.go:317] The initial operator extension status is healthy I0311 06:25:57.421801 1 controller.go:203] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0311 06:25:57.421808 1 controller.go:203] Source periodic-conditional *controllerstatus.Simple is not ready I0311 06:25:57.421811 1 controller.go:203] Source periodic-workloads *controllerstatus.Simple is not ready I0311 06:25:57.421824 1 controller.go:458] The operator is still being initialized I0311 06:25:57.421831 1 controller.go:481] The operator is healthy I0311 06:25:57.421876 1 tasks_processing.go:71] worker 10 working on service_accounts task. I0311 06:25:57.422137 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0311 06:25:57.422148 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0311 06:25:57.422436 1 operator.go:287] started I0311 06:25:57.422623 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0311 06:25:57.423737 1 recorder.go:70] Recording config/oauth with fingerprint=fcb1bd6f4c02948dfb56111218ba925d67d30765afce190d6e0fe2f73d0c441a I0311 06:25:57.423757 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 10.197161ms to process 1 records I0311 06:25:57.423801 1 tasks_processing.go:71] worker 3 working on schedulers task. I0311 06:25:57.423915 1 recorder.go:70] Recording config/apiserver with fingerprint=b3739981fdce58226e5f4b95c33e7eb8037e6bc3e6f62e1a715dbf87d55ac999 I0311 06:25:57.423926 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 11.167975ms to process 1 records I0311 06:25:57.424025 1 tasks_processing.go:71] worker 20 working on openstack_controlplanes task. I0311 06:25:57.424138 1 recorder.go:70] Recording config/authentication with fingerprint=4caff04e8d174dc15daafa0160c55f32e5b59edd3ed079354cae72c522ed1434 I0311 06:25:57.424149 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 11.496448ms to process 1 records I0311 06:25:57.424213 1 recorder.go:70] Recording config/image with fingerprint=f61cb9d5bd5e4f1fdd3182f513441c5113412e8c94ab39cfb07e24bee1e43b39 I0311 06:25:57.424225 1 gather.go:177] gatherer "clusterconfig" function "image" took 11.693436ms to process 1 records I0311 06:25:57.424240 1 tasks_processing.go:71] worker 19 working on aggregated_monitoring_cr_names task. I0311 06:25:57.426523 1 tasks_processing.go:71] worker 25 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0311 06:25:57.426665 1 recorder.go:70] Recording config/proxy with fingerprint=100a51ed1526abd7916f5bf9679ae74add439c44c1d7f4373f4b42a6818feda5 I0311 06:25:57.426685 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 14.732455ms to process 1 records I0311 06:25:57.426738 1 tasks_processing.go:71] worker 15 working on openstack_dataplanedeployments task. E0311 06:25:57.426747 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0311 06:25:57.426758 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 15.406616ms to process 0 records I0311 06:25:57.435628 1 tasks_processing.go:71] worker 22 working on sap_pods task. I0311 06:25:57.435635 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 16.912023ms to process 0 records I0311 06:25:57.436066 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 16.33366ms to process 0 records I0311 06:25:57.436250 1 tasks_processing.go:71] worker 30 working on networks task. I0311 06:25:57.437835 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 16.797956ms to process 0 records I0311 06:25:57.437848 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 11.82267ms to process 0 records I0311 06:25:57.437858 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 20.645359ms to process 0 records I0311 06:25:57.437865 1 tasks_processing.go:71] worker 20 working on machines task. I0311 06:25:57.437827 1 tasks_processing.go:71] worker 18 working on image_pruners task. I0311 06:25:57.437945 1 tasks_processing.go:71] worker 17 working on machine_config_pools task. I0311 06:25:57.438205 1 tasks_processing.go:71] worker 21 working on clusterroles task. I0311 06:25:57.438315 1 recorder.go:70] Recording config/storage/storageclasses/gp2-csi with fingerprint=4dcbe05f6ee0ff4e357441af6fd22b13df1d2a782fb11344dc4ec71ed2cdaa8a I0311 06:25:57.438644 1 recorder.go:70] Recording config/storage/storageclasses/gp3-csi with fingerprint=2659329ac4b16fa197d314aa3dde666cb5d4dc22d68dd05632daef133208f2a6 I0311 06:25:57.438764 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 27.086756ms to process 2 records I0311 06:25:57.438828 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 21.793955ms to process 0 records I0311 06:25:57.438860 1 tasks_processing.go:71] worker 7 working on crds task. I0311 06:25:57.438955 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 12.198005ms to process 0 records I0311 06:25:57.439005 1 tasks_processing.go:71] worker 15 working on cost_management_metrics_configs task. I0311 06:25:57.439709 1 tasks_processing.go:71] worker 8 working on ingress task. I0311 06:25:57.439866 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 27.480169ms to process 0 records I0311 06:25:57.440556 1 controller.go:203] Source scaController *sca.Controller is not ready I0311 06:25:57.440629 1 controller.go:203] Source clusterTransferController *clustertransfer.Controller is not ready I0311 06:25:57.440663 1 controller.go:203] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0311 06:25:57.440694 1 controller.go:203] Source periodic-conditional *controllerstatus.Simple is not ready I0311 06:25:57.440721 1 controller.go:203] Source periodic-workloads *controllerstatus.Simple is not ready I0311 06:25:57.440762 1 controller.go:458] The operator is still being initialized I0311 06:25:57.440791 1 controller.go:481] The operator is healthy I0311 06:25:57.440892 1 tasks_processing.go:71] worker 3 working on certificate_signing_requests task. I0311 06:25:57.441174 1 recorder.go:70] Recording config/schedulers/cluster with fingerprint=4b4937b52b6d6e10bc0afc918ffc02612f098c889a757815bf3d94154be706ac I0311 06:25:57.441224 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 17.075023ms to process 1 records I0311 06:25:57.441406 1 recorder.go:70] Recording config/featuregate with fingerprint=d0e9cb8f139b6d74b772e1c7d8a2d55609c4398e81eb700443bc7723374ca984 I0311 06:25:57.441426 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 22.268676ms to process 1 records I0311 06:25:57.441481 1 tasks_processing.go:71] worker 12 working on mutating_webhook_configurations task. I0311 06:25:57.441602 1 recorder.go:70] Recording config/pdbs/openshift-console/console with fingerprint=5408693df2e044b8e16a7f25c2624efdf76fcfcbfae9e4940aefd3abdd964dd8 I0311 06:25:57.441671 1 recorder.go:70] Recording config/pdbs/openshift-console/downloads with fingerprint=70a994490797519eb438d650e8ed19b54bbbb6cc9738b834aab6e5520ed53dd7 I0311 06:25:57.441716 1 recorder.go:70] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=ab98564e1f40c3a31ce40a428bdc7d53d43ecb8c93c33311bd41ba0bfdc0bd35 I0311 06:25:57.441772 1 recorder.go:70] Recording config/pdbs/openshift-ingress/router-default with fingerprint=cc6f851f3649fc228fa821ed0b80fbe5e19faa1ff1327833d2a95749c4ee34b6 I0311 06:25:57.441807 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=f9238519362b4e110e601cca7fd784d4b2fe242967d19af5e0894c80cbc04a92 I0311 06:25:57.441815 1 tasks_processing.go:71] worker 9 working on openshift_logging task. I0311 06:25:57.441834 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=449671d63f737551d92d427223ec419d538ccfadbc6974e75b942063efcb2f4e I0311 06:25:57.441899 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=4d56e716a6c34340311363fdde784fe0758f5af358c770acc62e2f27c507c1ab I0311 06:25:57.441965 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=2b3dd6f34b309462c9f19cf211b4601244ed285a27bd9deb634da00ebed4430b I0311 06:25:57.441998 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=7c611f53c4ea8de5be1785bf7a8aff342beda60ec08b664c9603e18daa4e239c I0311 06:25:57.442058 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=20863011cab369e665189308dc88953317db6222b00da28e42287ecff7f25f12 I0311 06:25:57.442109 1 recorder.go:70] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=295cfe77dfcaa6ae65e2f535bf5632fb4ea9425edd44e2f3374bf4a2512fd6d7 I0311 06:25:57.442142 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 29.398561ms to process 11 records I0311 06:25:57.442158 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 30.2679ms to process 0 records I0311 06:25:57.442165 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 22.229363ms to process 0 records I0311 06:25:57.442261 1 tasks_processing.go:71] worker 14 working on machine_healthchecks task. I0311 06:25:57.442314 1 tasks_processing.go:71] worker 26 working on storage_cluster task. I0311 06:25:57.442382 1 tasks_processing.go:74] worker 23 stopped. I0311 06:25:57.442692 1 recorder.go:70] Recording config/node/ip-10-0-0-117.ec2.internal with fingerprint=5791ca8b794b7384d2de5844b695a1b0e29ff9dd44e206b54d0d4adef324a50f I0311 06:25:57.442829 1 recorder.go:70] Recording config/node/ip-10-0-1-123.ec2.internal with fingerprint=fbb22c42dd74c0198f330c8083d9a4d9620fa7fbe6fbf0d1a2538953389778d4 I0311 06:25:57.442946 1 recorder.go:70] Recording config/node/ip-10-0-2-178.ec2.internal with fingerprint=38de3e1a934993129b8fb4a35efae499fd74cb1aa5eadab3e2a86a16ee80d808 I0311 06:25:57.442960 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 29.143827ms to process 3 records I0311 06:25:57.444182 1 tasks_processing.go:74] worker 6 stopped. W0311 06:25:57.444300 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0311 06:25:57.446873 1 recorder.go:70] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=bb400d3cd58fcd8ba64ff928e2979428a012d844cf0cb192c158c3686ae1adad I0311 06:25:57.446950 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 24.341307ms to process 1 records I0311 06:25:57.447617 1 tasks_processing.go:74] worker 4 stopped. I0311 06:25:57.448152 1 recorder.go:70] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=b3496872c61de1f681522be3fe50779290bb125466af56c0b483d16685b3ac26 I0311 06:25:57.448236 1 recorder.go:70] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=198de867f9bb7d85d3b97e9f1324a487ca9cec755383294083b832f8e88c4ebf I0311 06:25:57.448361 1 recorder.go:70] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=98e1c9a1342fb1b2ef7efa69f8f2dbda697f3fad1823552d4d8dd295f3a86762 I0311 06:25:57.448451 1 recorder.go:70] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=127ae2c9fbe504bee723a77eaf5e0d5b2c6d93b6f1753e322eb6a4549db6c90d I0311 06:25:57.448514 1 recorder.go:70] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=3c25da44e9eff1a377573ef5dfab4b8c7730a1415a0a9e8ed4068e37e0370986 I0311 06:25:57.448612 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=86c0207f933a756f13c924b493db3adbd15ac42ccb8627bedfa8c36d53fa2e31 I0311 06:25:57.448697 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=a42fa09b239ee0c88bec06ad8f56c8aa63950bc531a4646a5f2a694ffeb398db I0311 06:25:57.448793 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=3a4e78448ebde5fb190b7f10e6b02f8329ac349814380d3a3109cfb83d498fb6 I0311 06:25:57.448887 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=17c3f38e8abaae9b275aa2b717ff28e1dde54892e571af8bc1756540ce188df9 I0311 06:25:57.448965 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=99310436e929f0f0dd3bc8df2c9aa513b462e3a6fd6f37dbcf95fef91f2d433b I0311 06:25:57.449034 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=001e711ccf543d3316b597478ddba0c07a12f3593e9133663281525a12f3741b I0311 06:25:57.449117 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=afe272d7921e4f8ad7ce6aa4bb294d71cb9b507f1bea647230f9a652c9aeaf85 I0311 06:25:57.449152 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 28.604392ms to process 12 records I0311 06:25:57.449182 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 10.596534ms to process 0 records I0311 06:25:57.449209 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 8.291832ms to process 0 records I0311 06:25:57.449278 1 tasks_processing.go:74] worker 22 stopped. I0311 06:25:57.449280 1 tasks_processing.go:74] worker 17 stopped. I0311 06:25:57.449330 1 recorder.go:70] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0311 06:25:57.449383 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 32.910565ms to process 1 records E0311 06:25:57.449400 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0311 06:25:57.449409 1 gather.go:177] gatherer "clusterconfig" function "machines" took 11.019123ms to process 0 records I0311 06:25:57.449417 1 tasks_processing.go:74] worker 20 stopped. I0311 06:25:57.449336 1 tasks_processing.go:74] worker 28 stopped. I0311 06:25:57.449454 1 tasks_processing.go:74] worker 18 stopped. I0311 06:25:57.449634 1 recorder.go:70] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=68760367308bd4a4693218fb08553a881c9b2c919f246ef85594a5daf64bc9cf I0311 06:25:57.449688 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 11.557819ms to process 1 records I0311 06:25:57.449723 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 10.651821ms to process 0 records I0311 06:25:57.449767 1 tasks_processing.go:74] worker 15 stopped. I0311 06:25:57.449942 1 recorder.go:70] Recording config/network with fingerprint=1548d4ae73cd39aeeeeca4ac6f26571ab01f894758e71008016333f51c0c33f8 I0311 06:25:57.449987 1 gather.go:177] gatherer "clusterconfig" function "networks" took 11.830819ms to process 1 records I0311 06:25:57.450015 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 7.935088ms to process 0 records I0311 06:25:57.450046 1 tasks_processing.go:74] worker 9 stopped. I0311 06:25:57.450060 1 tasks_processing.go:74] worker 30 stopped. I0311 06:25:57.451333 1 tasks_processing.go:74] worker 2 stopped. I0311 06:25:57.452183 1 recorder.go:70] Recording config/version with fingerprint=5311e071372751183db483e2e6b7fea6780a54dcc372b1955058752a5d899c7c I0311 06:25:57.452245 1 recorder.go:70] Recording config/id with fingerprint=cc3aefef28aeaf75e314865484a660f4fb8020688f2b3e63046e229759800fc7 I0311 06:25:57.452262 1 gather.go:177] gatherer "clusterconfig" function "version" took 40.481739ms to process 2 records I0311 06:25:57.452932 1 tasks_processing.go:74] worker 5 stopped. I0311 06:25:57.452991 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 34.33445ms to process 0 records I0311 06:25:57.455186 1 tasks_processing.go:74] worker 8 stopped. I0311 06:25:57.455662 1 recorder.go:70] Recording config/ingress with fingerprint=a836405a05c91b6ee61fcc6becedb0f59528f8c092b1b262a3b8330a0503f26c I0311 06:25:57.455699 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 15.39495ms to process 1 records I0311 06:25:57.455819 1 tasks_processing.go:74] worker 12 stopped. I0311 06:25:57.455927 1 recorder.go:70] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=33af8e320d8bad6b511dac017c97b4dc384aedd84cdaa3f8b2c048b159671a05 I0311 06:25:57.456449 1 recorder.go:70] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=0471350f097aeaef5f315180edcb2e5863f931e0a8ef79e20357a508e8cdd418 I0311 06:25:57.456549 1 recorder.go:70] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=94597b6c635d3997e6ca05c633de4801f0ad5363b4001bb40e747e166d8841f2 I0311 06:25:57.456622 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 13.770247ms to process 3 records E0311 06:25:57.456641 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0311 06:25:57.456651 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 13.102453ms to process 0 records I0311 06:25:57.456662 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 13.063431ms to process 0 records I0311 06:25:57.456672 1 tasks_processing.go:74] worker 26 stopped. I0311 06:25:57.456678 1 tasks_processing.go:74] worker 14 stopped. I0311 06:25:57.456873 1 tasks_processing.go:74] worker 16 stopped. I0311 06:25:57.456986 1 recorder.go:70] Recording config/olm_operators with fingerprint=677275af4c6629cc0c8b567faf6df46ed838dc81c49e84913ca1f5e3493c7bae I0311 06:25:57.457010 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 45.504673ms to process 1 records I0311 06:25:57.459161 1 tasks_processing.go:74] worker 7 stopped. I0311 06:25:57.459973 1 recorder.go:70] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=32beaa8261403bdd4f6666d4bbbcd5a726cc21fcdf2c8afd5fbdd7f2897ac5f3 I0311 06:25:57.460316 1 recorder.go:70] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=7275a5c2f63e4eca29e7d8ac9d17d15cf96efd6c439be4e9304f0a6e821371ed I0311 06:25:57.460333 1 gather.go:177] gatherer "clusterconfig" function "crds" took 20.270559ms to process 2 records I0311 06:25:57.460344 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 19.058487ms to process 0 records I0311 06:25:57.460353 1 tasks_processing.go:74] worker 3 stopped. I0311 06:25:57.462532 1 tasks_processing.go:74] worker 1 stopped. I0311 06:25:57.464330 1 recorder.go:70] Recording config/pod/openshift-console-operator/console-operator-85b84f584d-vm2l2 with fingerprint=78e8a4676d8e9406d14f0682d1984cadebbda92f44fe5292dc3eadc67e422283 I0311 06:25:57.464908 1 recorder.go:70] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-fcgvr with fingerprint=0732ced52b2b4e5a9dde1d33a83d995db0a36f967410c895ee86cc4dcc76b421 I0311 06:25:57.465153 1 recorder.go:70] Recording config/running_containers with fingerprint=b97d17066557a5ae3dfe24e88617b24b94b4bb199f25bc4edab9f9f0d0803597 I0311 06:25:57.465199 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 51.100731ms to process 3 records I0311 06:25:57.465518 1 recorder.go:70] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=984e0f248481dd2f49b3045a6eb3db6f8836fc2842965e2af26baff27c386032 I0311 06:25:57.465603 1 tasks_processing.go:74] worker 21 stopped. I0311 06:25:57.465695 1 recorder.go:70] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=d40c7a1c38709d2539b037d61a36a8c69bfc6649a0b8fab414e53b8277748d43 I0311 06:25:57.465746 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 26.144959ms to process 2 records I0311 06:25:57.465783 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 41.217928ms to process 0 records I0311 06:25:57.465811 1 tasks_processing.go:74] worker 19 stopped. I0311 06:25:57.467975 1 tasks_processing.go:74] worker 29 stopped. E0311 06:25:57.467994 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0311 06:25:57.468004 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2ovbeu6i8bpkjb7e0df1muj9ermnd94m-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2ovbeu6i8bpkjb7e0df1muj9ermnd94m-primary-cert-bundle-secret" not found I0311 06:25:57.468097 1 recorder.go:70] Recording aggregated/ingress_controllers_certs with fingerprint=bd15c54cf369d82a967b047e234b3915afc520722dd7021ba60ec0bcd9b338e4 I0311 06:25:57.468124 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 57.062879ms to process 1 records I0311 06:25:57.471606 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0311 06:25:57.472818 1 tasks_processing.go:74] worker 11 stopped. E0311 06:25:57.472835 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0311 06:25:57.472844 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0311 06:25:57.472850 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0311 06:25:57.472862 1 recorder.go:70] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=295a2bc7810c628251f1f312c209b5eb4da888326706c635ac0c12aec0969e17 I0311 06:25:57.472896 1 recorder.go:70] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0311 06:25:57.472905 1 recorder.go:70] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=fe7abfa7f7aea852d8bca6b7df9b4d5de32045254ff2774f9bd30f0b6dcb7dc4 I0311 06:25:57.472910 1 recorder.go:70] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0311 06:25:57.472920 1 recorder.go:70] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0311 06:25:57.472963 1 recorder.go:70] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0311 06:25:57.472974 1 recorder.go:70] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0311 06:25:57.472982 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 61.001093ms to process 7 records I0311 06:25:57.473624 1 tasks_processing.go:74] worker 25 stopped. I0311 06:25:57.473638 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 47.080734ms to process 0 records I0311 06:25:57.484049 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0311 06:25:57.484055 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0311 06:25:57.484130 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0311 06:25:57.486976 1 base_controller.go:82] Caches are synced for ConfigController I0311 06:25:57.486989 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0311 06:25:57.491049 1 configmapobserver.go:84] configmaps "insights-config" not found I0311 06:25:57.523447 1 base_controller.go:82] Caches are synced for LoggingSyncer I0311 06:25:57.523463 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0311 06:25:57.626587 1 sca.go:266] etc-pki-entitlement secret successfully created I0311 06:25:57.626605 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0311 06:25:58.197685 1 cluster_transfer.go:102] no available accepted cluster transfer I0311 06:25:58.197707 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0311 06:25:58.274814 1 gather_cluster_operators.go:184] Unable to get operatorpkis.network.operator.openshift.io resource due to: operatorpkis.network.operator.openshift.io "ovn" is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot get resource "operatorpkis" in API group "network.operator.openshift.io" in the namespace "openshift-ovn-kubernetes" W0311 06:25:58.441767 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0311 06:25:58.467864 1 tasks_processing.go:74] worker 31 stopped. I0311 06:25:58.467916 1 recorder.go:70] Recording config/alerts with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 W0311 06:25:58.467931 1 gather.go:155] issue recording gatherer "clusterconfig" function "active_alerts" result "config/alerts.json" because of the warning: warning: the record with the same fingerprint "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945" was already recorded at path "config/namespaces_with_overlapping_uids.json", recording another one with a different path "config/alerts.json" I0311 06:25:58.467943 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.055840355s to process 1 records I0311 06:25:58.467957 1 recorder.go:70] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0311 06:25:58.467963 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.046378829s to process 1 records I0311 06:25:58.467969 1 tasks_processing.go:74] worker 13 stopped. I0311 06:25:58.474808 1 gather_cluster_operators.go:184] Unable to get operatorpkis.network.operator.openshift.io resource due to: operatorpkis.network.operator.openshift.io "signer" is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot get resource "operatorpkis" in API group "network.operator.openshift.io" in the namespace "openshift-ovn-kubernetes" W0311 06:25:59.441365 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0311 06:25:59.878961 1 tasks_processing.go:74] worker 27 stopped. I0311 06:25:59.879097 1 recorder.go:70] Recording config/clusteroperator/console with fingerprint=17d42f44acfd06dd6591e96bdde448b718212835e204015bb7823479ebf6ea5b I0311 06:25:59.879155 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0311 06:25:59.879199 1 recorder.go:70] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=46f1689ad7969e7bf328b69f7c0a3e0ecb8e7484ba3065367af1662e43c1014a I0311 06:25:59.879217 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0311 06:25:59.879249 1 recorder.go:70] Recording config/clusteroperator/dns with fingerprint=4caa13ed11644b0db89a5902e64c66e09458f6d4e12d891086c3e183311ce57e I0311 06:25:59.879277 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0311 06:25:59.879311 1 recorder.go:70] Recording config/clusteroperator/image-registry with fingerprint=f8d8a1e2fe07fcaa61bc271ebf50ec2adb45efe3c55b114a0f6cccf7c82e354d I0311 06:25:59.879349 1 recorder.go:70] Recording config/clusteroperator/ingress with fingerprint=30149b7f035a26ed57c1a6a8c56e92d48d2101b0d26ed3655124197a76436cfc I0311 06:25:59.879385 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=891c4b707b80fee6eec0e75131c3b5b40dbc55f809045a6164593190e44a3646 I0311 06:25:59.879427 1 recorder.go:70] Recording config/clusteroperator/insights with fingerprint=37fed25a3e4ec3e92b802fe8188dbed0c3743b9d98793dc60ad878f4c5d8a3f1 I0311 06:25:59.879732 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=8d05741102f07f5b71b2e91fd725083de7768f86c6998d9c7e46ccce8ed6c99f I0311 06:25:59.879768 1 recorder.go:70] Recording config/clusteroperator/kube-apiserver with fingerprint=5b081e320052562142c3b4c46b7236e52289fdb9ac25ad149726b1470a52015f I0311 06:25:59.879786 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0311 06:25:59.879814 1 recorder.go:70] Recording config/clusteroperator/kube-controller-manager with fingerprint=e435ecf0a58ba42c0c34ffc946fc34580a66823fd7651e003dd0d6ffba4d44b3 I0311 06:25:59.879833 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0311 06:25:59.879863 1 recorder.go:70] Recording config/clusteroperator/kube-scheduler with fingerprint=2fd3bd2b2efd57a30dab0404151513f9ee3828f3d1d0043ec3fefa95855de8ab I0311 06:25:59.879881 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0311 06:25:59.879912 1 recorder.go:70] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=c6c32c345720b7d95a73f064e1b1488de97c8b9e80298c472b28383a787b888d I0311 06:25:59.879929 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0311 06:25:59.879959 1 recorder.go:70] Recording config/clusteroperator/monitoring with fingerprint=9e5958a94d745e1615001e5da5b315047cac052c5be00d17e4718f82e51a3be7 I0311 06:25:59.880070 1 recorder.go:70] Recording config/clusteroperator/network with fingerprint=42c2c5ab299d136148d35eaf6d5ca22cd9ea3b0c8328632d1528305022832ca7 I0311 06:25:59.880105 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0311 06:25:59.880137 1 recorder.go:70] Recording config/clusteroperator/node-tuning with fingerprint=3e4950a0d97d3ae86d9cce926b211ec2391bdf368aa7badb63870454cd307e73 I0311 06:25:59.880172 1 recorder.go:70] Recording config/clusteroperator/openshift-apiserver with fingerprint=0dea2805e294b3c3dc9053b298910ec08a637d776bea1d469516f9e2ef8cf8a3 I0311 06:25:59.880188 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0311 06:25:59.880214 1 recorder.go:70] Recording config/clusteroperator/openshift-controller-manager with fingerprint=f58e2c14592f28f86ea0beb92d477fb440e75d1fc2815b116415a53c2bf1d4df I0311 06:25:59.880230 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0311 06:25:59.880258 1 recorder.go:70] Recording config/clusteroperator/openshift-samples with fingerprint=a8d09990fcd95a3af3ea0f0000a6fdaa8e684d6adaa78cff778a1310c37b6080 I0311 06:25:59.880277 1 recorder.go:70] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0311 06:25:59.880300 1 recorder.go:70] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=76f0f3049d583cbe139d7334b48149a8e1171eb040be731fbafa40add77109a3 I0311 06:25:59.880327 1 recorder.go:70] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=7431df644ae30901decb13057129943fb7bcddabb0321de61a0929c952e492ca I0311 06:25:59.880355 1 recorder.go:70] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=ee11c9b74465c53ed166af3e89b9446f4c405b029928922f0efcab857b768b20 I0311 06:25:59.880391 1 recorder.go:70] Recording config/clusteroperator/service-ca with fingerprint=6d46ca39be1914a4fa81036d10b1f60ac84e87b6fedf03df0a832b7d7bc5ea80 I0311 06:25:59.880408 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0311 06:25:59.880445 1 recorder.go:70] Recording config/clusteroperator/storage with fingerprint=40013c0952ea542f9c348ef7fc39495e97e833b1c2aac06cc6e8347e3b910234 I0311 06:25:59.880473 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0311 06:25:59.880488 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0311 06:25:59.880500 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.468110545s to process 37 records W0311 06:26:00.441251 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0311 06:26:01.441223 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0311 06:26:02.439265 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0311 06:26:02.439292 1 tasks_processing.go:74] worker 24 stopped. E0311 06:26:02.439303 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0311 06:26:02.439317 1 recorder.go:70] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0311 06:26:02.439327 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.028435674s to process 1 records I0311 06:26:10.252141 1 tasks_processing.go:74] worker 0 stopped. I0311 06:26:10.252201 1 recorder.go:70] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0311 06:26:10.252215 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.840012332s to process 1 records I0311 06:26:11.027517 1 tasks_processing.go:74] worker 10 stopped. I0311 06:26:11.027841 1 recorder.go:70] Recording config/serviceaccounts with fingerprint=73582af525bf551f979ef96bfb3cbb0257826668d63659ccecb8b26ded60e307 I0311 06:26:11.027862 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.605618261s to process 1 records E0311 06:26:11.027934 1 periodic.go:250] "Unhandled Error" err="clusterconfig failed after 13.617s with: function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_configs\" failed with an error, function \"support_secret\" failed with an error, function \"machines\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error" I0311 06:26:11.029050 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "tsdb_status" failed with an error, function "metrics" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_configs" failed with an error, function "support_secret" failed with an error, function "machines" failed with an error, function "machine_healthchecks" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error I0311 06:26:11.029069 1 periodic.go:212] Running workloads gatherer I0311 06:26:11.029095 1 tasks_processing.go:45] number of workers: 2 I0311 06:26:11.029112 1 tasks_processing.go:69] worker 1 listening for tasks. I0311 06:26:11.029120 1 tasks_processing.go:71] worker 1 working on helmchart_info task. I0311 06:26:11.029121 1 tasks_processing.go:69] worker 0 listening for tasks. I0311 06:26:11.029211 1 tasks_processing.go:71] worker 0 working on workload_info task. I0311 06:26:11.035459 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-123.ec2.internal... I0311 06:26:11.035471 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-178.ec2.internal... I0311 06:26:11.035482 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-117.ec2.internal... I0311 06:26:11.055979 1 tasks_processing.go:74] worker 1 stopped. I0311 06:26:11.055997 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 26.845206ms to process 0 records I0311 06:26:15.833483 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 5s I0311 06:26:15.857721 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0311 06:26:15.865290 1 gather_workloads_info.go:387] No image sha256:a7c71f3c9714cf1717c45e12ec817be6d6209f00989e6a31d634e527f0c8147d (9ms) I0311 06:26:15.873556 1 gather_workloads_info.go:387] No image sha256:bae7a33f8db8a3d4b3c4c05498aba85a0ce463f85322067c48e79663710e616e (8ms) I0311 06:26:15.885651 1 gather_workloads_info.go:387] No image sha256:929a4a9c96a08b57fe4c315deca55c30d1990c83e34b8b5692f7646d29f715c8 (12ms) I0311 06:26:15.897736 1 gather_workloads_info.go:387] No image sha256:00f977c2275f5bf03d423911159a39ca44fabc1862ecb7ae492198ec8c46c6bb (12ms) I0311 06:26:15.908261 1 gather_workloads_info.go:387] No image sha256:8b1e32ce43eb1247b0558230b6e0baa85ac62f1a5e2a2089d40b4fea90538529 (11ms) I0311 06:26:15.916517 1 gather_workloads_info.go:387] No image sha256:0a8a0473029ace3adbb66f490dbe560f0e7782a38cdf32c1f7dd3e092e1d191e (8ms) I0311 06:26:15.924835 1 gather_workloads_info.go:387] No image sha256:bcca5c17371caa2dce7e28da4c9fe30163c340fa172187f94db90ea1bd6ab80d (8ms) I0311 06:26:15.933255 1 gather_workloads_info.go:398] Found image sha256:59db91eadf1e1e5343777be43ff25fb712f300f9341be48ba05f45270944536f (8ms) I0311 06:26:15.942170 1 gather_workloads_info.go:387] No image sha256:527ed405bf66234718321996dacf51f91c64c746de9f1509416ef6ce89c90a50 (9ms) I0311 06:26:15.950759 1 gather_workloads_info.go:387] No image sha256:ce3d8211c0bde5788a4fd04c3eff31ae591437ceab2b3168dbbdfe408673c8b1 (9ms) I0311 06:26:15.966162 1 gather_workloads_info.go:387] No image sha256:46c2c396ca651699738c3f26b79b10af2922123dd27c2e32382d2a31a87e92c7 (15ms) I0311 06:26:16.067056 1 gather_workloads_info.go:387] No image sha256:4f175ee49f51dc4379e9993fecd1657c7a9c4c3dc096772cb198d0212b9eea47 (101ms) I0311 06:26:16.166260 1 gather_workloads_info.go:387] No image sha256:1cc2f3c83068c8c67d9070f67677e1bb6b4feff7ac5744cad08711893e56b55f (99ms) I0311 06:26:16.265996 1 gather_workloads_info.go:387] No image sha256:14703f73bb1ca69ea03b726e340b3b68f8e294e1f22c80d1a16666ea3d4a88a3 (100ms) I0311 06:26:16.366672 1 gather_workloads_info.go:387] No image sha256:5cba745f09f3bd6b8028c6773597cbb30bf3639a84f2b4b6493fe20642339a2c (101ms) I0311 06:26:16.466908 1 gather_workloads_info.go:387] No image sha256:87efe06afd1f04426fca7f86c0f74c4ee75c311ba199ddabd5b849b877bc59fa (100ms) I0311 06:26:16.566354 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (99ms) I0311 06:26:16.665975 1 gather_workloads_info.go:387] No image sha256:68d8e92046969bcfcbca2d2b3dcf4dcad5f6edc5fcb44861eb22f8079a43ea4d (100ms) I0311 06:26:16.766953 1 gather_workloads_info.go:387] No image sha256:9697bc2258bdfa9ae8c1866cc7eb0b3b46851998db827b121bdf77417a881eb3 (101ms) I0311 06:26:16.866715 1 gather_workloads_info.go:387] No image sha256:a3d34d8f38a540ce79d34a5248b1965346155497db57b72cb4758d659418059c (100ms) I0311 06:26:16.966248 1 gather_workloads_info.go:387] No image sha256:ccdda82b4adfa8ee4a84b3bad19693be6a42a9881a7b83788d835912129c49a4 (100ms) I0311 06:26:17.067407 1 gather_workloads_info.go:387] No image sha256:7854174b4b98e9b10d9c8e9b43113df1217ab21db4b2395d9399af0c3d7911a3 (101ms) I0311 06:26:17.166389 1 gather_workloads_info.go:387] No image sha256:7941b5d9a758b8667e99cd9236f7f96ec036af61df5cbc96e16077d36700d7c7 (99ms) I0311 06:26:17.266381 1 gather_workloads_info.go:387] No image sha256:420c12f61c53e54eab2d99476759c0de339fca98ce0a3a782bc6545cc0e97a9c (100ms) I0311 06:26:17.366272 1 gather_workloads_info.go:387] No image sha256:824a3b8f78e19aa21a9f6444aefe8e0b624886cca18fe828b13acbab55e6e868 (100ms) I0311 06:26:17.466174 1 gather_workloads_info.go:387] No image sha256:944d9261ba7a143131fe8267c172defc5f37acc1cea3d4d373ec6fc5d8bfcc31 (100ms) I0311 06:26:17.567408 1 gather_workloads_info.go:387] No image sha256:039479f355d6903e1018abf8427fece3465c4348d4de2144ec38114b51bec27e (101ms) I0311 06:26:17.666625 1 gather_workloads_info.go:387] No image sha256:2d077e73dd76873ee2a1583aa3b3e76a1b408737f0af2e22b2aa055604d89e81 (99ms) I0311 06:26:17.766544 1 gather_workloads_info.go:387] No image sha256:8ef4bdf07dda423fab73484dbc66d527ce8ba8cc2d4f99210bc1bc24ea08c0cb (100ms) I0311 06:26:17.866757 1 gather_workloads_info.go:387] No image sha256:a9e885d6f0456a2cc10f9e5da71fe9403f5e4c639b9b7ea15bd403b272ccb824 (100ms) I0311 06:26:17.966716 1 gather_workloads_info.go:387] No image sha256:38dd36da63b1f01316a02daf6c18b821318c6ec313712147292d74abf955e0da (100ms) I0311 06:26:18.066227 1 gather_workloads_info.go:387] No image sha256:2fac754deaeade3456361eed52e344318ff16d04819384432759f0ea35530114 (99ms) I0311 06:26:18.165477 1 gather_workloads_info.go:387] No image sha256:18956d5fbc8d0369a26cb1e00eb5c3aff1973ae8cbd3983920a0cf2851f05358 (99ms) I0311 06:26:18.265716 1 gather_workloads_info.go:387] No image sha256:1f2aa1e1026570162984f7fbb634635bcf319e408cafa8f9034ee842a2f193ab (100ms) I0311 06:26:18.366323 1 gather_workloads_info.go:387] No image sha256:0a92dd43975f972e3dc707fb37854e046813531553038147e9c90d54a8d9df73 (101ms) I0311 06:26:18.466061 1 gather_workloads_info.go:387] No image sha256:6c3c7f77d7a3815bbb69cad5f6f1742c17d9597bc135ba3d116c9736d98fae33 (100ms) I0311 06:26:18.566816 1 gather_workloads_info.go:387] No image sha256:799f5628c18e77a49c08645c03179046b5200c8cbcfd65737fb59668132f393e (101ms) I0311 06:26:18.667130 1 gather_workloads_info.go:398] Found image sha256:23c4fe84047a4e6cbe7f75f470e7d6fe0e61e7910dd17945b1b61bc4b72f3f2a (100ms) I0311 06:26:18.767137 1 gather_workloads_info.go:387] No image sha256:6277dac65decba0ea094146bb99600f479eaf804696842e724de1f1b436f9a38 (100ms) I0311 06:26:18.867431 1 gather_workloads_info.go:387] No image sha256:752be69b2262be713df12c47f4bac8c2dafed272c401e9a89f8060f053d68054 (100ms) I0311 06:26:18.967148 1 gather_workloads_info.go:387] No image sha256:e40bbab1e52f580ab471bd7ab19ae23a4bcbaf1893243a05238af6bede7ba9cc (100ms) I0311 06:26:19.066886 1 gather_workloads_info.go:387] No image sha256:373614619b9420b110d1508c5f17e066ce69c4c226fd04d02fbb959d9ba41eb6 (100ms) I0311 06:26:19.166635 1 gather_workloads_info.go:387] No image sha256:48404bd61d05dc738ccce4d22e36b30dbcdc6015b06b4afb604fd1baeee35bf2 (100ms) I0311 06:26:19.266216 1 gather_workloads_info.go:387] No image sha256:6d6ebff54b8adac74f4d1b12ac8aa0f16cd7b28370e0d6aa847d1c457e03a5b6 (100ms) I0311 06:26:19.365708 1 gather_workloads_info.go:387] No image sha256:76f820bd9bd138d29305d545d7d49bfe63e2923f1fb1ec2e8eac81a388359024 (99ms) I0311 06:26:19.465595 1 gather_workloads_info.go:387] No image sha256:3d0634fe58641d5242649c44ddba70ed67fdd6d2dcb5c2261df5cee8b33de9fd (100ms) I0311 06:26:19.465640 1 tasks_processing.go:74] worker 0 stopped. I0311 06:26:19.466333 1 recorder.go:70] Recording config/workload_info with fingerprint=5ce30c886a59adc1aef62d38c3e082fdaa193e4931590fffed77a279a2383318 I0311 06:26:19.466410 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.436416796s to process 1 records I0311 06:26:19.466434 1 periodic.go:259] Periodic gather workloads completed in 8.437s I0311 06:26:19.466447 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0311 06:26:19.466454 1 periodic.go:212] Running conditional gatherer I0311 06:26:19.474660 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.19.9/gathering_rules I0311 06:26:19.825773 1 conditional_gatherer.go:336] updating alerts cache for conditional gatherer E0311 06:26:19.828746 1 conditional_gatherer.go:322] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp 172.30.121.212:9091: connect: connection refused I0311 06:26:19.828832 1 conditional_gatherer.go:382] updating version cache for conditional gatherer I0311 06:26:19.837396 1 conditional_gatherer.go:390] cluster version is '4.19.9' E0311 06:26:19.837416 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 06:26:19.837425 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 06:26:19.837431 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 06:26:19.837436 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 06:26:19.837441 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 06:26:19.837446 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 06:26:19.837451 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 06:26:19.837456 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 06:26:19.837461 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0311 06:26:19.837467 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing I0311 06:26:19.837486 1 tasks_processing.go:45] number of workers: 3 I0311 06:26:19.837500 1 tasks_processing.go:69] worker 2 listening for tasks. I0311 06:26:19.837506 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0311 06:26:19.837513 1 tasks_processing.go:69] worker 0 listening for tasks. I0311 06:26:19.837523 1 tasks_processing.go:69] worker 1 listening for tasks. I0311 06:26:19.837533 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0311 06:26:19.837535 1 tasks_processing.go:74] worker 1 stopped. I0311 06:26:19.837571 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0311 06:26:19.837593 1 recorder.go:70] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0311 06:26:19.837615 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.316µs to process 1 records I0311 06:26:19.837709 1 tasks_processing.go:74] worker 2 stopped. I0311 06:26:19.837728 1 recorder.go:70] Recording insights-operator/conditional-gatherer-rules with fingerprint=9fafdcbf373f1f367f1d69551ea59114592cce1681189be0f3dde774e0530661 I0311 06:26:19.837742 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.544µs to process 1 records I0311 06:26:19.837866 1 tasks_processing.go:74] worker 0 stopped. I0311 06:26:19.837878 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 326.483µs to process 0 records I0311 06:26:19.837894 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=AsExpected message= I0311 06:26:19.837918 1 recorder.go:70] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0311 06:26:19.863045 1 recorder.go:70] Recording insights-operator/gathers with fingerprint=d55827d7ce3e5c723ad1d2450fd7a51102753d83123b2e5ae3129e3fab1f9fea I0311 06:26:19.863223 1 diskrecorder.go:70] Writing 109 records to /var/lib/insights-operator/insights-2026-03-11-062619.tar.gz I0311 06:26:19.871415 1 diskrecorder.go:51] Wrote 109 records to disk in 8ms I0311 06:26:19.871464 1 periodic.go:281] Gathering cluster info every 2h0m0s I0311 06:26:19.871486 1 periodic.go:282] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0311 06:27:27.411668 1 diskrecorder.go:223] Found files to send: insights-2026-03-11-062619.tar.gz I0311 06:27:27.411702 1 insightsuploader.go:150] Checking archives to upload periodically every 16m13.71505161s I0311 06:27:27.411711 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0311 06:27:27.425400 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0311 06:27:27.554954 1 requests.go:87] Successfully reported id=2026-03-11T06:27:27Z x-rh-insights-request-id=a9a504d0e6ee4ddc97b3e68a09ac4727, wrote=53030 I0311 06:27:27.555081 1 insightsuploader.go:187] Uploaded report successfully in 143.361942ms I0311 06:27:27.555105 1 controller.go:119] Initializing last reported time to 2026-03-11T06:27:27Z I0311 06:27:27.555177 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0311 06:27:27.555190 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0311 06:27:27.555199 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0311 06:27:27.560945 1 controller.go:481] The operator is healthy I0311 06:27:57.428794 1 controller.go:481] The operator is healthy I0311 06:28:33.281801 1 insightsreport.go:137] Pulling report from smart-proxy I0311 06:28:33.281850 1 insightsreport.go:149] Retrieving report I0311 06:28:33.292075 1 requests.go:110] Retrieving report for cluster: 28857ac8-334b-4db7-bfb8-855a066d7be5 I0311 06:28:33.292097 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/28857ac8-334b-4db7-bfb8-855a066d7be5/reports I0311 06:28:33.295296 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/28857ac8-334b-4db7-bfb8-855a066d7be5/reports I0311 06:28:33.493108 1 insightsreport.go:184] Report retrieved I0311 06:28:33.518498 1 insightsreport.go:239] Report retrieved correctly I0311 06:29:57.428807 1 controller.go:481] The operator is healthy I0311 06:30:57.404598 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0311 06:30:57.410214 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0311 06:30:57.410240 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0311 06:30:57.414278 1 secretconfigobserver.go:119] support secret does not exist I0311 06:31:57.495234 1 controller.go:481] The operator is healthy I0311 06:33:57.428892 1 controller.go:481] The operator is healthy I0311 06:35:57.404761 1 configmapobserver.go:84] configmaps "insights-config" not found I0311 06:35:57.414795 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0311 06:35:57.419656 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0311 06:35:57.419675 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0311 06:35:57.423697 1 secretconfigobserver.go:119] support secret does not exist I0311 06:35:57.427653 1 controller.go:481] The operator is healthy I0311 06:37:57.428482 1 controller.go:481] The operator is healthy I0311 06:39:57.429589 1 controller.go:481] The operator is healthy I0311 06:40:57.423966 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0311 06:40:57.430657 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0311 06:40:57.430678 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0311 06:40:57.434406 1 secretconfigobserver.go:119] support secret does not exist I0311 06:41:57.429495 1 controller.go:481] The operator is healthy I0311 06:43:46.272163 1 insightsuploader.go:146] Nothing to report since 2026-03-11T06:27:27Z I0311 06:43:57.430697 1 controller.go:481] The operator is healthy I0311 06:45:57.407137 1 configmapobserver.go:84] configmaps "insights-config" not found I0311 06:45:57.411534 1 configmapobserver.go:84] configmaps "insights-config" not found I0311 06:45:57.428321 1 controller.go:481] The operator is healthy I0311 06:45:57.435145 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0311 06:45:57.439599 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0311 06:45:57.439616 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0311 06:45:57.443348 1 secretconfigobserver.go:119] support secret does not exist I0311 06:47:57.430327 1 controller.go:481] The operator is healthy I0311 06:49:57.430509 1 controller.go:481] The operator is healthy I0311 06:50:57.443837 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0311 06:50:57.449173 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0311 06:50:57.449192 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0311 06:50:57.453096 1 secretconfigobserver.go:119] support secret does not exist I0311 06:51:57.429846 1 controller.go:481] The operator is healthy I0311 06:53:57.430841 1 controller.go:481] The operator is healthy I0311 06:55:57.406661 1 configmapobserver.go:84] configmaps "insights-config" not found I0311 06:55:57.428305 1 controller.go:481] The operator is healthy I0311 06:55:57.453329 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0311 06:55:57.457761 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0311 06:55:57.457781 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0311 06:55:57.461688 1 secretconfigobserver.go:119] support secret does not exist I0311 06:57:57.428988 1 controller.go:481] The operator is healthy I0311 06:59:20.122377 1 insightsuploader.go:146] Nothing to report since 2026-03-11T06:27:27Z I0311 06:59:57.429439 1 controller.go:481] The operator is healthy I0311 07:00:57.462177 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0311 07:00:57.468135 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0311 07:00:57.468158 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0311 07:00:57.472339 1 secretconfigobserver.go:119] support secret does not exist I0311 07:01:57.428913 1 controller.go:481] The operator is healthy I0311 07:03:57.429776 1 controller.go:481] The operator is healthy