I0312 16:08:24.149640 1 cmd.go:253] Using service-serving-cert provided certificates I0312 16:08:24.150109 1 observer_polling.go:159] Starting file observer W0312 16:08:24.170294 1 builder.go:272] unable to get owner reference (falling back to namespace): replicasets.apps "insights-operator-5ff5cb4f99" is forbidden: User "system:serviceaccount:openshift-insights:operator" cannot get resource "replicasets" in API group "apps" in the namespace "openshift-insights" I0312 16:08:24.523150 1 operator.go:59] Starting insights-operator v0.0.0-master+$Format:%H$ I0312 16:08:24.523504 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0312 16:08:24.524483 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0312 16:08:24.524545 1 secure_serving.go:57] Forcing use of http/1.1 only W0312 16:08:24.524570 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0312 16:08:24.524576 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0312 16:08:24.524584 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0312 16:08:24.524588 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0312 16:08:24.524593 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0312 16:08:24.524597 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0312 16:08:24.528547 1 operator.go:124] FeatureGates initialized: knownFeatureGates=[AWSEFSDriverVolumeMetrics AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BareMetalLoadBalancer BuildCSIVolumes CPMSMachineNamePrefix ChunkSizeMiB CloudDualStackNodeIPs ConsolePluginContentSecurityPolicy DisableKubeletCloudCredentialProviders ExternalOIDC GCPLabelsTags GatewayAPI GatewayAPIController HardwareSpeed IngressControllerLBSubnetsAWS KMSv1 ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles MultiArchInstallAWS MultiArchInstallGCP NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation NodeDisruptionPolicy OnClusterBuild PersistentIPsForVirtualization PrivateHostedZoneAWS RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController VSphereDriverConfiguration VSphereMultiVCenters ValidatingAdmissionPolicy AWSClusterHostedDNS AutomatedEtcdBackup BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNS GCPCustomAPIEndpoints HighlyAvailableArbiter ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather InsightsRuntimeExtractor KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController MachineAPIProviderOpenStack MachineConfigNodes MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PinnedImages PlatformOperators ProcMountType RouteAdvertisements SELinuxChangePolicy SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerification SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass VolumeGroupSnapshot] I0312 16:08:24.528588 1 event.go:377] Event(v1.ObjectReference{Kind:"Namespace", Namespace:"openshift-insights", Name:"openshift-insights", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AWSEFSDriverVolumeMetrics", "AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BareMetalLoadBalancer", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ChunkSizeMiB", "CloudDualStackNodeIPs", "ConsolePluginContentSecurityPolicy", "DisableKubeletCloudCredentialProviders", "ExternalOIDC", "GCPLabelsTags", "GatewayAPI", "GatewayAPIController", "HardwareSpeed", "IngressControllerLBSubnetsAWS", "KMSv1", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "MultiArchInstallAWS", "MultiArchInstallGCP", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "NodeDisruptionPolicy", "OnClusterBuild", "PersistentIPsForVirtualization", "PrivateHostedZoneAWS", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "VSphereDriverConfiguration", "VSphereMultiVCenters", "ValidatingAdmissionPolicy"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AutomatedEtcdBackup", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "HighlyAvailableArbiter", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "InsightsRuntimeExtractor", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "MachineAPIProviderOpenStack", "MachineConfigNodes", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PinnedImages", "PlatformOperators", "ProcMountType", "RouteAdvertisements", "SELinuxChangePolicy", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0312 16:08:24.530522 1 secure_serving.go:213] Serving securely on [::]:8443 I0312 16:08:24.530623 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0312 16:08:24.530656 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0312 16:08:24.530694 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0312 16:08:24.530824 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0312 16:08:24.530844 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0312 16:08:24.530696 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0312 16:08:24.531261 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0312 16:08:24.531296 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file E0312 16:08:24.532142 1 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:serviceaccount:openshift-insights:operator\" cannot create resource \"events\" in API group \"\" in the namespace \"openshift-insights\"" event="&Event{ObjectMeta:{openshift-insights.189c23ca2d5c42fb openshift-insights 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Namespace,Namespace:openshift-insights,Name:openshift-insights,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:FeatureGatesInitialized,Message:FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{\"AWSEFSDriverVolumeMetrics\", \"AdditionalRoutingCapabilities\", \"AdminNetworkPolicy\", \"AlibabaPlatform\", \"AzureWorkloadIdentity\", \"BareMetalLoadBalancer\", \"BuildCSIVolumes\", \"CPMSMachineNamePrefix\", \"ChunkSizeMiB\", \"CloudDualStackNodeIPs\", \"ConsolePluginContentSecurityPolicy\", \"DisableKubeletCloudCredentialProviders\", \"ExternalOIDC\", \"GCPLabelsTags\", \"GatewayAPI\", \"GatewayAPIController\", \"HardwareSpeed\", \"IngressControllerLBSubnetsAWS\", \"KMSv1\", \"ManagedBootImages\", \"ManagedBootImagesAWS\", \"MetricsCollectionProfiles\", \"MultiArchInstallAWS\", \"MultiArchInstallGCP\", \"NetworkDiagnosticsConfig\", \"NetworkLiveMigration\", \"NetworkSegmentation\", \"NodeDisruptionPolicy\", \"OnClusterBuild\", \"PersistentIPsForVirtualization\", \"PrivateHostedZoneAWS\", \"RouteExternalCertificate\", \"ServiceAccountTokenNodeBinding\", \"SetEIPForNLBIngressController\", \"VSphereDriverConfiguration\", \"VSphereMultiVCenters\", \"ValidatingAdmissionPolicy\"}, Disabled:[]v1.FeatureGateName{\"AWSClusterHostedDNS\", \"AutomatedEtcdBackup\", \"BootcNodeManagement\", \"ClusterAPIInstall\", \"ClusterAPIInstallIBMCloud\", \"ClusterMonitoringConfig\", \"ClusterVersionOperatorConfiguration\", \"DNSNameResolver\", \"DualReplica\", \"DyanmicServiceEndpointIBMCloud\", \"DynamicResourceAllocation\", \"EtcdBackendQuota\", \"EventedPLEG\", \"Example\", \"Example2\", \"ExternalOIDCWithUIDAndExtraClaimMappings\", \"GCPClusterHostedDNS\", \"GCPCustomAPIEndpoints\", \"HighlyAvailableArbiter\", \"ImageStreamImportMode\", \"IngressControllerDynamicConfigurationManager\", \"InsightsConfig\", \"InsightsConfigAPI\", \"InsightsOnDemandDataGather\", \"InsightsRuntimeExtractor\", \"KMSEncryptionProvider\", \"MachineAPIMigration\", \"MachineAPIOperatorDisableMachineHealthCheckController\", \"MachineAPIProviderOpenStack\", \"MachineConfigNodes\", \"MaxUnavailableStatefulSet\", \"MinimumKubeletVersion\", \"MixedCPUsAllocation\", \"MultiArchInstallAzure\", \"NewOLM\", \"NewOLMCatalogdAPIV1Metas\", \"NewOLMOwnSingleNamespace\", \"NewOLMPreflightPermissionChecks\", \"NodeSwap\", \"NutanixMultiSubnets\", \"OVNObservability\", \"OpenShiftPodSecurityAdmission\", \"PinnedImages\", \"PlatformOperators\", \"ProcMountType\", \"RouteAdvertisements\", \"SELinuxChangePolicy\", \"SELinuxMount\", \"ShortCertRotation\", \"SignatureStores\", \"SigstoreImageVerification\", \"SigstoreImageVerificationPKI\", \"TranslateStreamCloseWebsocketRequests\", \"UpgradeStatus\", \"UserNamespacesPodSecurityStandards\", \"UserNamespacesSupport\", \"VSphereConfigurableMaxAllowedBlockVolumesPerNode\", \"VSphereHostVMGroupZonal\", \"VSphereMultiDisk\", \"VSphereMultiNetworks\", \"VolumeAttributesClass\", \"VolumeGroupSnapshot\"}},Source:EventSource{Component:openshift-insights-operator,Host:,},FirstTimestamp:2026-03-12 16:08:24.528519931 +0000 UTC m=+0.424205886,LastTimestamp:2026-03-12 16:08:24.528519931 +0000 UTC m=+0.424205886,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:openshift-insights-operator,ReportingInstance:,}" W0312 16:08:24.534276 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0312 16:08:24.534303 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0312 16:08:24.534494 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0312 16:08:24.540569 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0312 16:08:24.540592 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0312 16:08:24.545466 1 secretconfigobserver.go:119] support secret does not exist I0312 16:08:24.550668 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0312 16:08:24.554863 1 secretconfigobserver.go:119] support secret does not exist I0312 16:08:24.556784 1 recorder.go:156] Pruning old reports every 5h58m50s, max age is 288h0m0s I0312 16:08:24.561822 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0312 16:08:24.561839 1 periodic.go:212] Running clusterconfig gatherer I0312 16:08:24.561860 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0312 16:08:24.561890 1 tasks_processing.go:45] number of workers: 32 I0312 16:08:24.561902 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0312 16:08:24.561914 1 insightsreport.go:296] Starting report retriever I0312 16:08:24.561921 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0312 16:08:24.561923 1 tasks_processing.go:69] worker 1 listening for tasks. I0312 16:08:24.561928 1 tasks_processing.go:69] worker 3 listening for tasks. I0312 16:08:24.561933 1 tasks_processing.go:69] worker 11 listening for tasks. I0312 16:08:24.561935 1 tasks_processing.go:69] worker 2 listening for tasks. I0312 16:08:24.561941 1 tasks_processing.go:69] worker 21 listening for tasks. I0312 16:08:24.561943 1 tasks_processing.go:71] worker 11 working on validating_webhook_configurations task. I0312 16:08:24.561944 1 tasks_processing.go:69] worker 22 listening for tasks. I0312 16:08:24.561950 1 tasks_processing.go:69] worker 12 listening for tasks. I0312 16:08:24.561950 1 tasks_processing.go:71] worker 21 working on mutating_webhook_configurations task. I0312 16:08:24.561953 1 tasks_processing.go:71] worker 22 working on cluster_apiserver task. I0312 16:08:24.561956 1 tasks_processing.go:69] worker 5 listening for tasks. I0312 16:08:24.561959 1 tasks_processing.go:69] worker 13 listening for tasks. I0312 16:08:24.561968 1 tasks_processing.go:69] worker 6 listening for tasks. I0312 16:08:24.561971 1 tasks_processing.go:69] worker 14 listening for tasks. I0312 16:08:24.561978 1 tasks_processing.go:69] worker 7 listening for tasks. I0312 16:08:24.561980 1 tasks_processing.go:69] worker 15 listening for tasks. I0312 16:08:24.561987 1 tasks_processing.go:69] worker 16 listening for tasks. I0312 16:08:24.561992 1 tasks_processing.go:69] worker 8 listening for tasks. I0312 16:08:24.561996 1 tasks_processing.go:69] worker 17 listening for tasks. I0312 16:08:24.562002 1 tasks_processing.go:69] worker 9 listening for tasks. I0312 16:08:24.562005 1 tasks_processing.go:69] worker 18 listening for tasks. I0312 16:08:24.561934 1 tasks_processing.go:71] worker 1 working on infrastructures task. I0312 16:08:24.562011 1 tasks_processing.go:69] worker 10 listening for tasks. I0312 16:08:24.562014 1 tasks_processing.go:69] worker 19 listening for tasks. I0312 16:08:24.562020 1 tasks_processing.go:69] worker 20 listening for tasks. I0312 16:08:24.562030 1 tasks_processing.go:69] worker 27 listening for tasks. I0312 16:08:24.562032 1 tasks_processing.go:69] worker 30 listening for tasks. I0312 16:08:24.562039 1 tasks_processing.go:69] worker 23 listening for tasks. I0312 16:08:24.562160 1 tasks_processing.go:71] worker 23 working on pod_network_connectivity_checks task. I0312 16:08:24.561944 1 tasks_processing.go:71] worker 2 working on overlapping_namespace_uids task. I0312 16:08:24.562043 1 tasks_processing.go:69] worker 28 listening for tasks. I0312 16:08:24.562383 1 tasks_processing.go:71] worker 28 working on feature_gates task. I0312 16:08:24.562052 1 tasks_processing.go:69] worker 29 listening for tasks. I0312 16:08:24.562066 1 tasks_processing.go:69] worker 25 listening for tasks. I0312 16:08:24.562074 1 tasks_processing.go:69] worker 24 listening for tasks. I0312 16:08:24.562085 1 tasks_processing.go:71] worker 3 working on metrics task. I0312 16:08:24.562444 1 tasks_processing.go:71] worker 24 working on image task. I0312 16:08:24.562466 1 tasks_processing.go:71] worker 29 working on storage_cluster task. I0312 16:08:24.562079 1 tasks_processing.go:69] worker 0 listening for tasks. I0312 16:08:24.562502 1 tasks_processing.go:71] worker 25 working on config_maps task. I0312 16:08:24.562599 1 tasks_processing.go:71] worker 0 working on operators task. I0312 16:08:24.562093 1 tasks_processing.go:71] worker 30 working on ceph_cluster task. I0312 16:08:24.562098 1 tasks_processing.go:71] worker 16 working on operators_pods_and_events task. I0312 16:08:24.562098 1 tasks_processing.go:71] worker 8 working on machine_autoscalers task. I0312 16:08:24.562104 1 tasks_processing.go:71] worker 12 working on authentication task. I0312 16:08:24.562105 1 tasks_processing.go:71] worker 17 working on clusterroles task. I0312 16:08:24.562108 1 tasks_processing.go:71] worker 9 working on machine_configs task. I0312 16:08:24.562111 1 tasks_processing.go:71] worker 5 working on image_registries task. I0312 16:08:24.562112 1 tasks_processing.go:71] worker 14 working on openstack_dataplanenodesets task. I0312 16:08:24.562113 1 tasks_processing.go:71] worker 18 working on lokistack task. I0312 16:08:24.562116 1 tasks_processing.go:71] worker 7 working on machine_config_pools task. I0312 16:08:24.562117 1 tasks_processing.go:71] worker 13 working on nodenetworkstates task. I0312 16:08:24.565281 1 tasks_processing.go:71] worker 23 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. E0312 16:08:24.565290 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0312 16:08:24.565650 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 3.110011ms to process 0 records I0312 16:08:24.562122 1 tasks_processing.go:71] worker 19 working on dvo_metrics task. I0312 16:08:24.562125 1 tasks_processing.go:71] worker 20 working on oauths task. I0312 16:08:24.562121 1 tasks_processing.go:71] worker 6 working on crds task. I0312 16:08:24.562129 1 tasks_processing.go:71] worker 27 working on pdbs task. I0312 16:08:24.561946 1 tasks_processing.go:69] worker 4 listening for tasks. I0312 16:08:24.562120 1 tasks_processing.go:71] worker 15 working on sap_datahubs task. I0312 16:08:24.561936 1 tasks_processing.go:69] worker 31 listening for tasks. I0312 16:08:24.562101 1 tasks_processing.go:69] worker 26 listening for tasks. I0312 16:08:24.562118 1 tasks_processing.go:71] worker 10 working on ingress task. I0312 16:08:24.566272 1 tasks_processing.go:71] worker 4 working on machine_healthchecks task. I0312 16:08:24.566540 1 tasks_processing.go:71] worker 31 working on storage_classes task. I0312 16:08:24.566797 1 tasks_processing.go:71] worker 26 working on ingress_certificates task. E0312 16:08:24.568041 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.252.181:9091: connect: connection refused I0312 16:08:24.568067 1 tasks_processing.go:71] worker 3 working on cost_management_metrics_configs task. E0312 16:08:24.568081 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.252.181:9091: connect: connection refused I0312 16:08:24.568115 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 5.622411ms to process 0 records I0312 16:08:24.570075 1 tasks_processing.go:71] worker 8 working on service_accounts task. I0312 16:08:24.570166 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 6.943839ms to process 0 records I0312 16:08:24.570189 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 7.57953ms to process 0 records I0312 16:08:24.570196 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 7.319587ms to process 0 records I0312 16:08:24.570237 1 tasks_processing.go:71] worker 29 working on container_images task. I0312 16:08:24.570250 1 tasks_processing.go:71] worker 30 working on support_secret task. I0312 16:08:24.570995 1 controller.go:119] Initializing last reported time to 0001-01-01T00:00:00Z I0312 16:08:24.571015 1 controller.go:317] The initial operator extension status is healthy I0312 16:08:24.571026 1 controller.go:203] Source periodic-conditional *controllerstatus.Simple is not ready I0312 16:08:24.571032 1 controller.go:203] Source periodic-workloads *controllerstatus.Simple is not ready I0312 16:08:24.571038 1 controller.go:203] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0312 16:08:24.571057 1 controller.go:458] The operator is still being initialized I0312 16:08:24.571068 1 controller.go:481] The operator is healthy I0312 16:08:24.571199 1 tasks_processing.go:71] worker 22 working on container_runtime_configs task. W0312 16:08:24.571311 1 operator.go:287] started I0312 16:08:24.571338 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0312 16:08:24.571505 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0312 16:08:24.571581 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0312 16:08:24.571842 1 recorder.go:70] Recording config/apiserver with fingerprint=0d5b532b52aec3d16598eb8477baeceac2350316ecf25e5c86a539cfab2d5388 I0312 16:08:24.571864 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 9.236781ms to process 1 records I0312 16:08:24.572038 1 tasks_processing.go:71] worker 1 working on jaegers task. I0312 16:08:24.572670 1 recorder.go:70] Recording config/infrastructure with fingerprint=abac98042f3b7b558cd5561aa44d3412535838a444185f4dcd1bb7a54584e586 I0312 16:08:24.572687 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 9.402764ms to process 1 records I0312 16:08:24.572764 1 tasks_processing.go:71] worker 24 working on node_logs task. I0312 16:08:24.572774 1 recorder.go:70] Recording config/image with fingerprint=6ba8b8339a1587bf4d4f7c447a259662835b9353bb24ec8c6cd4ff2177a0e40a I0312 16:08:24.572784 1 gather.go:177] gatherer "clusterconfig" function "image" took 9.143745ms to process 1 records I0312 16:08:24.577370 1 tasks_processing.go:71] worker 7 working on silenced_alerts task. I0312 16:08:24.577454 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 12.279791ms to process 0 records E0312 16:08:24.577511 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0312 16:08:24.577530 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 11.160267ms to process 0 records I0312 16:08:24.577557 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 12.707871ms to process 0 records I0312 16:08:24.577586 1 tasks_processing.go:71] worker 18 working on active_alerts task. I0312 16:08:24.577587 1 tasks_processing.go:71] worker 4 working on openstack_version task. I0312 16:08:24.577718 1 tasks_processing.go:71] worker 21 working on image_pruners task. I0312 16:08:24.577808 1 recorder.go:70] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=50d60b2e7e01791bf032c534a54dbf9a82368c2c27486749edb3f4b94d9700f9 I0312 16:08:24.577873 1 recorder.go:70] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=ac5a8a8903a257152c8e7a006554db6250eb4c8af4c1b865af6be866066c024c I0312 16:08:24.577937 1 recorder.go:70] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=5f603746db8a803f8c0065f14151a4b3e6ba32b7bdc8445e74a17b9826344058 I0312 16:08:24.577951 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 15.604936ms to process 3 records I0312 16:08:24.577961 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 9.510558ms to process 0 records I0312 16:08:24.577981 1 tasks_processing.go:71] worker 3 working on proxies task. I0312 16:08:24.578069 1 recorder.go:70] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=aec87b3e4fa8ecae2628752890a89202d3e1bf077fb6deec846278ed918e37f6 I0312 16:08:24.578138 1 tasks_processing.go:71] worker 11 working on olm_operators task. I0312 16:08:24.578143 1 recorder.go:70] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=69cfb82b82f1066b63bf36fbf055aae39d34850da1ed343a5a01359168b624e6 I0312 16:08:24.578489 1 recorder.go:70] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=bf6a307faa9035c410011c5b44b8ce351bec6759d14a1af1e388e92a6b169875 I0312 16:08:24.578895 1 recorder.go:70] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=b450733e3dac475ff6a68a3342a255d8bd340b0fc1277541dbc18b675407879e I0312 16:08:24.578938 1 recorder.go:70] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=fa1ef9b7db443be6be08f511b909f3155553dda9304906dc6bc7f2d46f1dc726 I0312 16:08:24.578990 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=f1824d3f6564525d797adec41ea07ecdc9b6e0ed4e4b7ab8968ff5863595cf56 I0312 16:08:24.579040 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=69023b8dbc12cb296f2548d415d3bd2cc72357d99d2d1306e2b7f663ad132695 I0312 16:08:24.579093 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=ed568b27417f4eb836510dcf2a4b18f0378e2afee352f3714372a42975586681 I0312 16:08:24.579165 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=3f517298c4ae0af48589e19eccfb610a7c4831621699f30bcb1cbfd56dd7ac3c I0312 16:08:24.579250 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=9a2a49eccdfe9a8dbbcbe455fd4d74c9ce583e7ad7e92601cb900ed57f986811 I0312 16:08:24.579305 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=ff6c0b076285e6ce9c4593724db8c07b3cf7a0a9be95c15f1e09c4fdf04d17b3 I0312 16:08:24.579394 1 recorder.go:70] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=ac09555fc581f70cbd06f9d5ea0b86441075aef4f3de2b5a24e083b46df5447d I0312 16:08:24.579409 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 15.738674ms to process 12 records I0312 16:08:24.579426 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 12.750239ms to process 0 records I0312 16:08:24.579433 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 13.476968ms to process 0 records I0312 16:08:24.579454 1 recorder.go:70] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0312 16:08:24.579501 1 tasks_processing.go:71] worker 2 working on machine_sets task. I0312 16:08:24.579508 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 16.237328ms to process 1 records I0312 16:08:24.579488 1 tasks_processing.go:71] worker 13 working on openstack_controlplanes task. I0312 16:08:24.579496 1 tasks_processing.go:71] worker 14 working on openstack_dataplanedeployments task. I0312 16:08:24.587626 1 tasks_processing.go:71] worker 15 working on nodenetworkconfigurationpolicies task. I0312 16:08:24.587641 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 21.000813ms to process 0 records I0312 16:08:24.587809 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 16.571705ms to process 0 records I0312 16:08:24.587829 1 tasks_processing.go:71] worker 22 working on tsdb_status task. I0312 16:08:24.588296 1 tasks_processing.go:71] worker 12 working on monitoring_persistent_volumes task. I0312 16:08:24.588621 1 recorder.go:70] Recording config/authentication with fingerprint=f98aa9a3e3b11ded68fb1f0078d27590f33ed02d591c133b2c32628a5b5d8810 I0312 16:08:24.588640 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 24.772838ms to process 1 records I0312 16:08:24.588731 1 tasks_processing.go:71] worker 20 working on sap_pods task. I0312 16:08:24.589475 1 recorder.go:70] Recording config/oauth with fingerprint=f2b7d38fc4b4eca476494262f30902df99dcba7691d343b2c2766ab18aca3973 I0312 16:08:24.589544 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 22.027997ms to process 1 records I0312 16:08:24.589834 1 recorder.go:70] Recording config/ingress with fingerprint=ece5c505a7eaefa6e9792ccf829da78109348679a66a3a64c502669bc2ce6e86 I0312 16:08:24.589876 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 22.650842ms to process 1 records I0312 16:08:24.589891 1 tasks_processing.go:71] worker 10 working on sap_config task. E0312 16:08:24.592271 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.252.181:9091: connect: connection refused I0312 16:08:24.592544 1 tasks_processing.go:71] worker 22 working on version task. E0312 16:08:24.592549 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.252.181:9091: connect: connection refused I0312 16:08:24.592650 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 4.635997ms to process 0 records I0312 16:08:24.595434 1 tasks_processing.go:71] worker 1 working on openshift_machine_api_events task. I0312 16:08:24.595755 1 controller.go:203] Source periodic-workloads *controllerstatus.Simple is not ready I0312 16:08:24.595780 1 controller.go:203] Source scaController *sca.Controller is not ready I0312 16:08:24.595787 1 controller.go:203] Source clusterTransferController *clustertransfer.Controller is not ready I0312 16:08:24.595793 1 controller.go:203] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0312 16:08:24.595800 1 controller.go:203] Source periodic-conditional *controllerstatus.Simple is not ready I0312 16:08:24.595820 1 controller.go:458] The operator is still being initialized I0312 16:08:24.595834 1 controller.go:481] The operator is healthy I0312 16:08:24.596006 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 23.344846ms to process 0 records I0312 16:08:24.596174 1 recorder.go:70] Recording config/pdbs/openshift-console/console with fingerprint=5a00e6a858651a447a3429af8672b1f0e97c9a70cea0d96e65423afd98450513 I0312 16:08:24.596245 1 recorder.go:70] Recording config/pdbs/openshift-console/downloads with fingerprint=92640ae843aa667899fdcb71d62ed24efa362b4b85c9b8499f5dbbe266faddf9 I0312 16:08:24.596272 1 recorder.go:70] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=a18a667a7aa715d118609d8e81c804a394fe6f0d8578aee6177eb9bf3e4669f3 I0312 16:08:24.596349 1 recorder.go:70] Recording config/pdbs/openshift-ingress/router-default with fingerprint=6b0c592f57291fe25c3074b3783fb6c09c492fff606590993d4d30b244d48568 I0312 16:08:24.596415 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=78fdaafa363c8e748c0e14030d7799140346523cca2e7c5cccd2e691acea1940 I0312 16:08:24.596280 1 tasks_processing.go:71] worker 27 working on schedulers task. I0312 16:08:24.596481 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=542f8d64308175ad3d01fe68befcbe7aa797ddb5c3e2e08b6f7793865ca08d0e I0312 16:08:24.596785 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=b6bf2472619e3a3734e53106869abedec0949e02150dfd884399140000de99d1 I0312 16:08:24.596845 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=f64bee618a380dff52b9894adc0f239c20e767e9234a0e92020d69b578379b22 I0312 16:08:24.596893 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=002978611a7b436f1d2870dbd443a9bfa733bcc1cbf3ec0f66ab4c54487c6f62 I0312 16:08:24.596953 1 recorder.go:70] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=e0c55b751fee1e4b6fa13d4667d057f00c7091c82006d3d21fb75e55088ca65d I0312 16:08:24.597004 1 recorder.go:70] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=72ac7d530a1ee6e068e6120c8ff5cce35a8cc6f2a93f0ee0ef4c97ed0d4736da I0312 16:08:24.597017 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 29.18089ms to process 11 records I0312 16:08:24.597125 1 recorder.go:70] Recording config/storage/storageclasses/gp2-csi with fingerprint=9f81d76b768018069ae2a5f987a4617d4a99f682c63822169774a916a16d6dd2 I0312 16:08:24.597172 1 recorder.go:70] Recording config/storage/storageclasses/gp3-csi with fingerprint=2e9216a8a3a3ba97da8a9c37bf8787f485463c841e274774f9572a708599867f I0312 16:08:24.597201 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 29.534298ms to process 2 records I0312 16:08:24.597261 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 23.546403ms to process 0 records I0312 16:08:24.597288 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 18.950247ms to process 0 records I0312 16:08:24.597314 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 17.126513ms to process 0 records I0312 16:08:24.597340 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 17.035654ms to process 0 records I0312 16:08:24.597367 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 17.189093ms to process 0 records I0312 16:08:24.597395 1 tasks_processing.go:71] worker 13 working on machines task. I0312 16:08:24.597416 1 tasks_processing.go:71] worker 31 working on aggregated_monitoring_cr_names task. I0312 16:08:24.597606 1 tasks_processing.go:71] worker 24 working on networks task. I0312 16:08:24.597406 1 tasks_processing.go:71] worker 14 working on certificate_signing_requests task. I0312 16:08:24.597413 1 tasks_processing.go:71] worker 4 working on openshift_logging task. I0312 16:08:24.597656 1 tasks_processing.go:71] worker 2 working on nodes task. I0312 16:08:24.598296 1 tasks_processing.go:71] worker 30 working on install_plans task. E0312 16:08:24.598783 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0312 16:08:24.598799 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 27.948999ms to process 0 records E0312 16:08:24.598808 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0312 16:08:24.598835 1 recorder.go:70] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0312 16:08:24.598849 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 34.328711ms to process 1 records I0312 16:08:24.599083 1 tasks_processing.go:74] worker 9 stopped. I0312 16:08:24.599267 1 recorder.go:70] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=c507923049730c9b6ea2e6db4b308cc32b7b7bf9ac8114922b29a41642309ee1 I0312 16:08:24.599324 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 21.049784ms to process 1 records I0312 16:08:24.599279 1 tasks_processing.go:74] worker 21 stopped. I0312 16:08:24.599458 1 tasks_processing.go:74] worker 10 stopped. I0312 16:08:24.599483 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 9.453141ms to process 0 records I0312 16:08:24.599539 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 10.722842ms to process 0 records I0312 16:08:24.599569 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 11.837413ms to process 0 records I0312 16:08:24.599598 1 tasks_processing.go:74] worker 15 stopped. I0312 16:08:24.599623 1 tasks_processing.go:74] worker 20 stopped. I0312 16:08:24.599843 1 tasks_processing.go:74] worker 28 stopped. I0312 16:08:24.601376 1 recorder.go:70] Recording config/featuregate with fingerprint=3ce2b3be993e4f87d7154f98f8d307aafaa53e8fbfd22e9185d9a995a204331c I0312 16:08:24.601398 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 37.445965ms to process 1 records I0312 16:08:24.601545 1 recorder.go:70] Recording config/proxy with fingerprint=b8ed1f663c8c1c21a4e3a570e4d17a7369eda1081d70f230eed08480a839584a I0312 16:08:24.601592 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 21.980916ms to process 1 records I0312 16:08:24.601908 1 tasks_processing.go:74] worker 3 stopped. I0312 16:08:24.602551 1 recorder.go:70] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=9d770b7c6532908b416bece6d5df21a94a273b22280b74296bad5b17f7c96261 I0312 16:08:24.602573 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 36.602409ms to process 1 records I0312 16:08:24.602583 1 tasks_processing.go:74] worker 5 stopped. I0312 16:08:24.605764 1 tasks_processing.go:74] worker 17 stopped. I0312 16:08:24.606242 1 recorder.go:70] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=3bbeb4a68a94a563c8f67f3eabe7f7db7569306f44220fde3999f0a840afa8c2 I0312 16:08:24.609062 1 recorder.go:70] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=951cc848709af228a15c1c00c49cf306e59a5c6dd331e84d05efb57c6886b1be I0312 16:08:24.609084 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 42.131666ms to process 2 records I0312 16:08:24.609108 1 recorder.go:70] Recording config/dvo_metrics with fingerprint=108780cb24f15bf572a4031966507398a0d0c6a28b3a008952f53b481dc191a9 I0312 16:08:24.609122 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 41.551801ms to process 1 records I0312 16:08:24.609133 1 tasks_processing.go:74] worker 19 stopped. I0312 16:08:24.609311 1 tasks_processing.go:74] worker 12 stopped. I0312 16:08:24.609341 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 20.9911ms to process 0 records I0312 16:08:24.610860 1 tasks_processing.go:74] worker 13 stopped. E0312 16:08:24.610878 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0312 16:08:24.610890 1 gather.go:177] gatherer "clusterconfig" function "machines" took 13.436366ms to process 0 records I0312 16:08:24.610999 1 tasks_processing.go:74] worker 4 stopped. I0312 16:08:24.611017 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 13.302523ms to process 0 records I0312 16:08:24.611074 1 tasks_processing.go:74] worker 1 stopped. I0312 16:08:24.611117 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 15.584203ms to process 0 records I0312 16:08:24.611410 1 tasks_processing.go:74] worker 6 stopped. I0312 16:08:24.612171 1 recorder.go:70] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=1f4a7d137ae1301fe1763445f07cc4286b1b57353052ba38c6d7cbbcbef7f075 I0312 16:08:24.612469 1 recorder.go:70] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=7e2c18a7864a00efd5713aefe93a730f52d7064c4e93fc630d4bbf97cf124d77 I0312 16:08:24.612486 1 gather.go:177] gatherer "clusterconfig" function "crds" took 45.134384ms to process 2 records I0312 16:08:24.613878 1 tasks_processing.go:74] worker 27 stopped. I0312 16:08:24.614032 1 recorder.go:70] Recording config/schedulers/cluster with fingerprint=9e758efb0a80e682a8101d508ea3f202a84e6801b01fd69bd7b198739ae1b0a8 I0312 16:08:24.614048 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 17.380407ms to process 1 records I0312 16:08:24.614182 1 tasks_processing.go:74] worker 24 stopped. I0312 16:08:24.614287 1 recorder.go:70] Recording config/network with fingerprint=dfac520aa19c025ef02fe7e44c8696b31fc7407df711b2e26288e09e5b34f831 I0312 16:08:24.614302 1 gather.go:177] gatherer "clusterconfig" function "networks" took 16.36553ms to process 1 records I0312 16:08:24.620749 1 tasks_processing.go:74] worker 2 stopped. I0312 16:08:24.621754 1 recorder.go:70] Recording config/node/ip-10-0-0-16.ec2.internal with fingerprint=7b52138b4c092169bbd2c02138887309b8379c9cfeb11dde5f3c5ffbaee2127c I0312 16:08:24.621954 1 recorder.go:70] Recording config/node/ip-10-0-1-71.ec2.internal with fingerprint=eb66b6577c2e4dc17c9ae16aae9c07ad002a79b8f3b4211c78edb11841a8f641 I0312 16:08:24.622068 1 recorder.go:70] Recording config/node/ip-10-0-2-99.ec2.internal with fingerprint=e7fffd1a75e0205e87d1d81b19a4d879f3613a2064b4984eeacf8924709bef08 I0312 16:08:24.622098 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 22.788744ms to process 3 records I0312 16:08:24.625176 1 tasks_processing.go:74] worker 14 stopped. I0312 16:08:24.625193 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 27.489646ms to process 0 records I0312 16:08:24.631263 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0312 16:08:24.631355 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0312 16:08:24.631495 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0312 16:08:24.634531 1 tasks_processing.go:74] worker 23 stopped. I0312 16:08:24.634562 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 69.1801ms to process 0 records I0312 16:08:24.634914 1 base_controller.go:82] Caches are synced for ConfigController I0312 16:08:24.634982 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0312 16:08:24.639338 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0312 16:08:24.641143 1 tasks_processing.go:74] worker 29 stopped. I0312 16:08:24.643982 1 recorder.go:70] Recording config/pod/openshift-console-operator/console-operator-85b84f584d-n5mvp with fingerprint=db5b472f1d7e1f4e0c7559d5e02f2cd2d4e45fab5b57af0a2b05d1e7c46ddfad I0312 16:08:24.644517 1 recorder.go:70] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-gwfns with fingerprint=467d715ba09a7a67dba9205c502332d64fa579d871a820b968c14a8b8d584401 I0312 16:08:24.644634 1 recorder.go:70] Recording config/running_containers with fingerprint=218fd0dacceee6acb3adaf5c0e393ec1e7adedea00637dbbea3702c24b517adf I0312 16:08:24.644655 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 70.883236ms to process 3 records I0312 16:08:24.644794 1 recorder.go:70] Recording config/olm_operators with fingerprint=ed7c007cf3e68e8d6e7b43f796a127441875f34b78904fe098d2fe5a39809979 I0312 16:08:24.644850 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 64.103305ms to process 1 records I0312 16:08:24.644864 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 45.635727ms to process 0 records E0312 16:08:24.644875 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0312 16:08:24.644882 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0312 16:08:24.644887 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0312 16:08:24.644898 1 recorder.go:70] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=295a2bc7810c628251f1f312c209b5eb4da888326706c635ac0c12aec0969e17 I0312 16:08:24.644800 1 tasks_processing.go:74] worker 11 stopped. I0312 16:08:24.644953 1 tasks_processing.go:74] worker 31 stopped. I0312 16:08:24.644942 1 recorder.go:70] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0312 16:08:24.645028 1 recorder.go:70] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=fe7abfa7f7aea852d8bca6b7df9b4d5de32045254ff2774f9bd30f0b6dcb7dc4 I0312 16:08:24.645040 1 recorder.go:70] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0312 16:08:24.645056 1 recorder.go:70] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0312 16:08:24.644949 1 tasks_processing.go:74] worker 25 stopped. I0312 16:08:24.645120 1 recorder.go:70] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0312 16:08:24.645176 1 recorder.go:70] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0312 16:08:24.645235 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 80.484094ms to process 7 records I0312 16:08:24.646054 1 configmapobserver.go:84] configmaps "insights-config" not found I0312 16:08:24.649426 1 tasks_processing.go:74] worker 26 stopped. E0312 16:08:24.649441 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0312 16:08:24.649448 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2p08kqbffbgkvb38d49p3vr3jtpagklm-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2p08kqbffbgkvb38d49p3vr3jtpagklm-primary-cert-bundle-secret" not found I0312 16:08:24.649521 1 recorder.go:70] Recording aggregated/ingress_controllers_certs with fingerprint=a7bdcb0f46d1d1cce1bd3325307507a16bbc9324b0d9bc853910452a64b4e7e6 I0312 16:08:24.649537 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 82.588975ms to process 1 records I0312 16:08:24.672342 1 base_controller.go:82] Caches are synced for LoggingSyncer I0312 16:08:24.672357 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0312 16:08:24.683120 1 gather_cluster_operator_pods_and_events.go:121] Found 0 pods with 0 containers I0312 16:08:24.683142 1 tasks_processing.go:74] worker 16 stopped. I0312 16:08:24.683151 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 120.212073ms to process 0 records I0312 16:08:24.686590 1 tasks_processing.go:74] worker 22 stopped. I0312 16:08:24.687093 1 recorder.go:70] Recording config/version with fingerprint=b2eaa50decc8e895e8ada9d1b81ea59c9af0f71d8433520552d69ede4af79010 I0312 16:08:24.687109 1 recorder.go:70] Recording config/id with fingerprint=e70fc7ad1fd88e5b344dc1c06bc00d1c93d74029d8a58fcadc74d6fb34defa25 I0312 16:08:24.687117 1 gather.go:177] gatherer "clusterconfig" function "version" took 93.994043ms to process 2 records I0312 16:08:24.715249 1 cluster_transfer.go:102] no available accepted cluster transfer I0312 16:08:24.715268 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0312 16:08:24.743552 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot get resource "dnsrecords" in API group "ingress.operator.openshift.io" in the namespace "openshift-ingress-operator" I0312 16:08:24.749484 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot get resource "dnsrecords" in API group "ingress.operator.openshift.io" in the namespace "openshift-ingress" I0312 16:08:24.826648 1 sca.go:266] etc-pki-entitlement secret successfully created I0312 16:08:24.826668 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0312 16:08:25.450909 1 gather_cluster_operators.go:184] Unable to get operatorpkis.network.operator.openshift.io resource due to: operatorpkis.network.operator.openshift.io "ovn" is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot get resource "operatorpkis" in API group "network.operator.openshift.io" in the namespace "openshift-ovn-kubernetes" I0312 16:08:25.649556 1 gather_cluster_operators.go:184] Unable to get operatorpkis.network.operator.openshift.io resource due to: operatorpkis.network.operator.openshift.io "signer" is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot get resource "operatorpkis" in API group "network.operator.openshift.io" in the namespace "openshift-ovn-kubernetes" I0312 16:08:25.666668 1 tasks_processing.go:74] worker 18 stopped. I0312 16:08:25.666722 1 recorder.go:70] Recording config/alerts with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 W0312 16:08:25.666739 1 gather.go:155] issue recording gatherer "clusterconfig" function "active_alerts" result "config/alerts.json" because of the warning: warning: the record with the same fingerprint "4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945" was already recorded at path "config/namespaces_with_overlapping_uids.json", recording another one with a different path "config/alerts.json" I0312 16:08:25.666754 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.089061892s to process 1 records I0312 16:08:25.666767 1 recorder.go:70] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0312 16:08:25.666773 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.089269098s to process 1 records I0312 16:08:25.666778 1 tasks_processing.go:74] worker 7 stopped. I0312 16:08:27.054048 1 tasks_processing.go:74] worker 0 stopped. I0312 16:08:27.054174 1 recorder.go:70] Recording config/clusteroperator/console with fingerprint=1e6f3465ab250a97f3d70eac75854a46e89e0cf863aae43f122f4535908e00c3 I0312 16:08:27.054247 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0312 16:08:27.054269 1 recorder.go:70] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=b9000e0a6d315a4a88ba679d6854f45bde6084bb5970244667a191b692ea8035 I0312 16:08:27.054279 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0312 16:08:27.054303 1 recorder.go:70] Recording config/clusteroperator/dns with fingerprint=f4e2dc4ad4cdd52a09740307a1ea341482ddd6963dff7ff2f8903f1a4e2ac332 I0312 16:08:27.054321 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0312 16:08:27.054342 1 recorder.go:70] Recording config/clusteroperator/image-registry with fingerprint=313f30207f964023d23b6bfc90c6ab81141e935f29c0048e27fa1787351e83e9 I0312 16:08:27.054369 1 recorder.go:70] Recording config/clusteroperator/ingress with fingerprint=a6d59191e0c59e303fe7bcd48523f5df51eac8e6b5c87def048c3a017390e541 I0312 16:08:27.054394 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=a8196990e7ccc40f8aeab64756b7c8c102c05d80e882aaf6ab878a449bceea75 I0312 16:08:27.054425 1 recorder.go:70] Recording config/clusteroperator/insights with fingerprint=6937bf4f4336243c749c763d0eb7f31de044e398fbedf35f80e9d4641bf58782 I0312 16:08:27.054708 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=d1427e0b2a885e568e5de7ee6c560462faec363c0a52b1ab47de1597494ec523 I0312 16:08:27.054729 1 recorder.go:70] Recording config/clusteroperator/kube-apiserver with fingerprint=e187f2acd3a77c0f6f3d02c1f68e5fd66b4a376d0940d914631e053a5fda12cb I0312 16:08:27.054739 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0312 16:08:27.054757 1 recorder.go:70] Recording config/clusteroperator/kube-controller-manager with fingerprint=3cc3da57315764efe46cf588697d0b5d0aa89396e3b7960fa8e8c84ebf478528 I0312 16:08:27.054769 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0312 16:08:27.054785 1 recorder.go:70] Recording config/clusteroperator/kube-scheduler with fingerprint=5b2310effb25cfd72bdebb73b2d0dd7b08121e8e219260555de94bf4c389becb I0312 16:08:27.054795 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0312 16:08:27.054815 1 recorder.go:70] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=875478fefa2504b00773030599477f3c79683fdf83269ee9486b87ef3906c114 I0312 16:08:27.054827 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0312 16:08:27.054848 1 recorder.go:70] Recording config/clusteroperator/monitoring with fingerprint=aa13a910d7bf7ecfed3d7d6524e53750d85898625754f775ce1ae09dfae987d2 I0312 16:08:27.054949 1 recorder.go:70] Recording config/clusteroperator/network with fingerprint=6715db12c679d8cad515ca0163c75b32fb8808c3b61774468cbeaae306707f17 I0312 16:08:27.054975 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0312 16:08:27.054997 1 recorder.go:70] Recording config/clusteroperator/node-tuning with fingerprint=379554399876c6625578d04ecabf2cc55abefcf24cfe18cfa0b8d761ed7892f9 I0312 16:08:27.055021 1 recorder.go:70] Recording config/clusteroperator/openshift-apiserver with fingerprint=abb46a98442682b4dee3286862320169bda6e5f25055601cddf63029a09203ba I0312 16:08:27.055030 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0312 16:08:27.055047 1 recorder.go:70] Recording config/clusteroperator/openshift-controller-manager with fingerprint=18cbf620425f10010b49be6d7f3da07311dc64d918a3ca40013b30f49f10a683 I0312 16:08:27.055058 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0312 16:08:27.055076 1 recorder.go:70] Recording config/clusteroperator/openshift-samples with fingerprint=acb70bb75c2fb57c36927c50d5dda259afa7f337b62e482ab1323fe265144c61 I0312 16:08:27.055089 1 recorder.go:70] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0312 16:08:27.055104 1 recorder.go:70] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=efd74cf89fb952f1f63e29d33303a36f04c0638009fa0d1d145fea0b7debc829 I0312 16:08:27.055119 1 recorder.go:70] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=33fa1e24364368372c70b8224eb77efa25ebb121232208ff14564de4218a3c77 I0312 16:08:27.055136 1 recorder.go:70] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=4f1f83d056c224a03a005ee3efdebf70ef071ebbf2c7113077726c469bd5fd75 I0312 16:08:27.055157 1 recorder.go:70] Recording config/clusteroperator/service-ca with fingerprint=097b5c29c1d6a56a743304afdb821db7b8b6aa6c448a2b6397d029e418ef1441 I0312 16:08:27.055168 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0312 16:08:27.055231 1 recorder.go:70] Recording config/clusteroperator/storage with fingerprint=b4ab9e39a217b42e0fe79e9a7a07db451f76b07db292cb77f9a0f9d0efac1c6a I0312 16:08:27.055251 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0312 16:08:27.055261 1 recorder.go:70] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0312 16:08:27.055269 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.491429734s to process 37 records I0312 16:08:37.484948 1 tasks_processing.go:74] worker 30 stopped. I0312 16:08:37.484995 1 recorder.go:70] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0312 16:08:37.485007 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.886627582s to process 1 records I0312 16:08:38.177780 1 tasks_processing.go:74] worker 8 stopped. I0312 16:08:38.178108 1 recorder.go:70] Recording config/serviceaccounts with fingerprint=73582af525bf551f979ef96bfb3cbb0257826668d63659ccecb8b26ded60e307 I0312 16:08:38.178130 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.607625156s to process 1 records E0312 16:08:38.178192 1 periodic.go:250] "Unhandled Error" err="clusterconfig failed after 13.616s with: function \"pod_network_connectivity_checks\" failed with an error, function \"metrics\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"tsdb_status\" failed with an error, function \"support_secret\" failed with an error, function \"machine_configs\" failed with an error, function \"machines\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error" I0312 16:08:38.179330 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "metrics" failed with an error, function "machine_healthchecks" failed with an error, function "tsdb_status" failed with an error, function "support_secret" failed with an error, function "machine_configs" failed with an error, function "machines" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error I0312 16:08:38.179347 1 periodic.go:212] Running workloads gatherer I0312 16:08:38.179364 1 tasks_processing.go:45] number of workers: 2 I0312 16:08:38.179373 1 tasks_processing.go:69] worker 1 listening for tasks. I0312 16:08:38.179377 1 tasks_processing.go:71] worker 1 working on helmchart_info task. I0312 16:08:38.179381 1 tasks_processing.go:69] worker 0 listening for tasks. I0312 16:08:38.179452 1 tasks_processing.go:71] worker 0 working on workload_info task. I0312 16:08:38.186130 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-99.ec2.internal... I0312 16:08:38.186143 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-16.ec2.internal... I0312 16:08:38.186158 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-71.ec2.internal... I0312 16:08:38.208261 1 tasks_processing.go:74] worker 1 stopped. I0312 16:08:38.208275 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 28.871901ms to process 0 records I0312 16:08:43.870685 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 6s I0312 16:08:43.896365 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0312 16:08:43.903885 1 gather_workloads_info.go:387] No image sha256:bae7a33f8db8a3d4b3c4c05498aba85a0ce463f85322067c48e79663710e616e (9ms) I0312 16:08:43.913572 1 gather_workloads_info.go:387] No image sha256:a7c71f3c9714cf1717c45e12ec817be6d6209f00989e6a31d634e527f0c8147d (10ms) I0312 16:08:43.921474 1 gather_workloads_info.go:387] No image sha256:8b1e32ce43eb1247b0558230b6e0baa85ac62f1a5e2a2089d40b4fea90538529 (8ms) I0312 16:08:43.929182 1 gather_workloads_info.go:387] No image sha256:2fac754deaeade3456361eed52e344318ff16d04819384432759f0ea35530114 (8ms) I0312 16:08:43.936596 1 gather_workloads_info.go:387] No image sha256:3d0634fe58641d5242649c44ddba70ed67fdd6d2dcb5c2261df5cee8b33de9fd (7ms) I0312 16:08:43.943935 1 gather_workloads_info.go:387] No image sha256:1f2aa1e1026570162984f7fbb634635bcf319e408cafa8f9034ee842a2f193ab (7ms) I0312 16:08:43.950942 1 gather_workloads_info.go:387] No image sha256:e40bbab1e52f580ab471bd7ab19ae23a4bcbaf1893243a05238af6bede7ba9cc (7ms) I0312 16:08:43.958283 1 gather_workloads_info.go:387] No image sha256:87efe06afd1f04426fca7f86c0f74c4ee75c311ba199ddabd5b849b877bc59fa (7ms) I0312 16:08:43.965580 1 gather_workloads_info.go:387] No image sha256:929a4a9c96a08b57fe4c315deca55c30d1990c83e34b8b5692f7646d29f715c8 (7ms) I0312 16:08:43.973271 1 gather_workloads_info.go:387] No image sha256:0a92dd43975f972e3dc707fb37854e046813531553038147e9c90d54a8d9df73 (8ms) I0312 16:08:44.003872 1 gather_workloads_info.go:387] No image sha256:4f175ee49f51dc4379e9993fecd1657c7a9c4c3dc096772cb198d0212b9eea47 (31ms) I0312 16:08:44.103877 1 gather_workloads_info.go:387] No image sha256:00f977c2275f5bf03d423911159a39ca44fabc1862ecb7ae492198ec8c46c6bb (100ms) I0312 16:08:44.204799 1 gather_workloads_info.go:398] Found image sha256:59db91eadf1e1e5343777be43ff25fb712f300f9341be48ba05f45270944536f (101ms) I0312 16:08:44.304549 1 gather_workloads_info.go:398] Found image sha256:23c4fe84047a4e6cbe7f75f470e7d6fe0e61e7910dd17945b1b61bc4b72f3f2a (100ms) I0312 16:08:44.404505 1 gather_workloads_info.go:387] No image sha256:a3d34d8f38a540ce79d34a5248b1965346155497db57b72cb4758d659418059c (100ms) I0312 16:08:44.504836 1 gather_workloads_info.go:387] No image sha256:2d077e73dd76873ee2a1583aa3b3e76a1b408737f0af2e22b2aa055604d89e81 (100ms) I0312 16:08:44.604150 1 gather_workloads_info.go:387] No image sha256:18956d5fbc8d0369a26cb1e00eb5c3aff1973ae8cbd3983920a0cf2851f05358 (99ms) I0312 16:08:44.704302 1 gather_workloads_info.go:387] No image sha256:6277dac65decba0ea094146bb99600f479eaf804696842e724de1f1b436f9a38 (100ms) I0312 16:08:44.804124 1 gather_workloads_info.go:387] No image sha256:1cc2f3c83068c8c67d9070f67677e1bb6b4feff7ac5744cad08711893e56b55f (100ms) I0312 16:08:44.904099 1 gather_workloads_info.go:387] No image sha256:38dd36da63b1f01316a02daf6c18b821318c6ec313712147292d74abf955e0da (100ms) I0312 16:08:45.004098 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (100ms) I0312 16:08:45.105522 1 gather_workloads_info.go:387] No image sha256:7854174b4b98e9b10d9c8e9b43113df1217ab21db4b2395d9399af0c3d7911a3 (101ms) I0312 16:08:45.204689 1 gather_workloads_info.go:387] No image sha256:a9e885d6f0456a2cc10f9e5da71fe9403f5e4c639b9b7ea15bd403b272ccb824 (99ms) I0312 16:08:45.305685 1 gather_workloads_info.go:387] No image sha256:ce3d8211c0bde5788a4fd04c3eff31ae591437ceab2b3168dbbdfe408673c8b1 (101ms) I0312 16:08:45.404935 1 gather_workloads_info.go:387] No image sha256:799f5628c18e77a49c08645c03179046b5200c8cbcfd65737fb59668132f393e (99ms) I0312 16:08:45.504063 1 gather_workloads_info.go:387] No image sha256:76f820bd9bd138d29305d545d7d49bfe63e2923f1fb1ec2e8eac81a388359024 (99ms) I0312 16:08:45.604536 1 gather_workloads_info.go:387] No image sha256:6d6ebff54b8adac74f4d1b12ac8aa0f16cd7b28370e0d6aa847d1c457e03a5b6 (100ms) I0312 16:08:45.704579 1 gather_workloads_info.go:387] No image sha256:6c3c7f77d7a3815bbb69cad5f6f1742c17d9597bc135ba3d116c9736d98fae33 (100ms) I0312 16:08:45.804428 1 gather_workloads_info.go:387] No image sha256:14703f73bb1ca69ea03b726e340b3b68f8e294e1f22c80d1a16666ea3d4a88a3 (100ms) I0312 16:08:45.904792 1 gather_workloads_info.go:387] No image sha256:1d471ad866b8564b209214d7dd3a7363c47e9535211d660b63b46abb278af7ab (100ms) I0312 16:08:46.005090 1 gather_workloads_info.go:387] No image sha256:752be69b2262be713df12c47f4bac8c2dafed272c401e9a89f8060f053d68054 (100ms) I0312 16:08:46.104734 1 gather_workloads_info.go:387] No image sha256:824a3b8f78e19aa21a9f6444aefe8e0b624886cca18fe828b13acbab55e6e868 (100ms) I0312 16:08:46.204408 1 gather_workloads_info.go:387] No image sha256:ce98d5d844bfc2ba8de1893866ad38166c95157d54abd8192b181e819bc50bb5 (100ms) I0312 16:08:46.304193 1 gather_workloads_info.go:387] No image sha256:8ef4bdf07dda423fab73484dbc66d527ce8ba8cc2d4f99210bc1bc24ea08c0cb (100ms) I0312 16:08:46.404022 1 gather_workloads_info.go:387] No image sha256:0a8a0473029ace3adbb66f490dbe560f0e7782a38cdf32c1f7dd3e092e1d191e (100ms) I0312 16:08:46.503443 1 gather_workloads_info.go:387] No image sha256:527ed405bf66234718321996dacf51f91c64c746de9f1509416ef6ce89c90a50 (99ms) I0312 16:08:46.603890 1 gather_workloads_info.go:387] No image sha256:bcca5c17371caa2dce7e28da4c9fe30163c340fa172187f94db90ea1bd6ab80d (100ms) I0312 16:08:46.705249 1 gather_workloads_info.go:387] No image sha256:9697bc2258bdfa9ae8c1866cc7eb0b3b46851998db827b121bdf77417a881eb3 (101ms) I0312 16:08:46.804864 1 gather_workloads_info.go:387] No image sha256:48404bd61d05dc738ccce4d22e36b30dbcdc6015b06b4afb604fd1baeee35bf2 (100ms) I0312 16:08:46.905311 1 gather_workloads_info.go:387] No image sha256:ccdda82b4adfa8ee4a84b3bad19693be6a42a9881a7b83788d835912129c49a4 (100ms) I0312 16:08:47.004137 1 gather_workloads_info.go:387] No image sha256:420c12f61c53e54eab2d99476759c0de339fca98ce0a3a782bc6545cc0e97a9c (99ms) I0312 16:08:47.105286 1 gather_workloads_info.go:387] No image sha256:7941b5d9a758b8667e99cd9236f7f96ec036af61df5cbc96e16077d36700d7c7 (101ms) I0312 16:08:47.204488 1 gather_workloads_info.go:387] No image sha256:373614619b9420b110d1508c5f17e066ce69c4c226fd04d02fbb959d9ba41eb6 (99ms) I0312 16:08:47.305036 1 gather_workloads_info.go:387] No image sha256:46c2c396ca651699738c3f26b79b10af2922123dd27c2e32382d2a31a87e92c7 (101ms) I0312 16:08:47.403596 1 gather_workloads_info.go:387] No image sha256:944d9261ba7a143131fe8267c172defc5f37acc1cea3d4d373ec6fc5d8bfcc31 (99ms) I0312 16:08:47.504895 1 gather_workloads_info.go:387] No image sha256:68d8e92046969bcfcbca2d2b3dcf4dcad5f6edc5fcb44861eb22f8079a43ea4d (101ms) I0312 16:08:47.604099 1 gather_workloads_info.go:387] No image sha256:039479f355d6903e1018abf8427fece3465c4348d4de2144ec38114b51bec27e (99ms) I0312 16:08:47.703883 1 gather_workloads_info.go:387] No image sha256:5cba745f09f3bd6b8028c6773597cbb30bf3639a84f2b4b6493fe20642339a2c (100ms) I0312 16:08:47.703921 1 tasks_processing.go:74] worker 0 stopped. I0312 16:08:47.704265 1 recorder.go:70] Recording config/workload_info with fingerprint=439a108dab8f36f74498240142d20e6307378fe76dfae516ccd664d25626c784 I0312 16:08:47.704289 1 gather.go:177] gatherer "workloads" function "workload_info" took 9.524457004s to process 1 records I0312 16:08:47.704303 1 periodic.go:259] Periodic gather workloads completed in 9.524s I0312 16:08:47.704310 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0312 16:08:47.704315 1 periodic.go:212] Running conditional gatherer I0312 16:08:47.712450 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.19.9/gathering_rules I0312 16:08:48.112862 1 conditional_gatherer.go:336] updating alerts cache for conditional gatherer E0312 16:08:48.115479 1 conditional_gatherer.go:322] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp 172.30.252.181:9091: connect: connection refused I0312 16:08:48.115566 1 conditional_gatherer.go:382] updating version cache for conditional gatherer I0312 16:08:48.124720 1 conditional_gatherer.go:390] cluster version is '4.19.9' E0312 16:08:48.124736 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0312 16:08:48.124741 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0312 16:08:48.124745 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0312 16:08:48.124749 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0312 16:08:48.124754 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0312 16:08:48.124757 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0312 16:08:48.124760 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0312 16:08:48.124763 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0312 16:08:48.124766 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing E0312 16:08:48.124769 1 conditional_gatherer.go:207] error checking conditions for a gathering rule: alerts cache is missing I0312 16:08:48.124782 1 tasks_processing.go:45] number of workers: 3 I0312 16:08:48.124789 1 tasks_processing.go:69] worker 2 listening for tasks. I0312 16:08:48.124793 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0312 16:08:48.124800 1 tasks_processing.go:69] worker 0 listening for tasks. I0312 16:08:48.124816 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0312 16:08:48.124821 1 tasks_processing.go:69] worker 1 listening for tasks. I0312 16:08:48.124838 1 tasks_processing.go:74] worker 1 stopped. I0312 16:08:48.124837 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0312 16:08:48.124867 1 recorder.go:70] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0312 16:08:48.124883 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 882ns to process 1 records I0312 16:08:48.124955 1 recorder.go:70] Recording insights-operator/conditional-gatherer-rules with fingerprint=9fafdcbf373f1f367f1d69551ea59114592cce1681189be0f3dde774e0530661 I0312 16:08:48.124967 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 2.493µs to process 1 records I0312 16:08:48.124971 1 tasks_processing.go:74] worker 2 stopped. I0312 16:08:48.125079 1 tasks_processing.go:74] worker 0 stopped. I0312 16:08:48.125090 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 257.096µs to process 0 records I0312 16:08:48.125105 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=AsExpected message= I0312 16:08:48.125123 1 recorder.go:70] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0312 16:08:48.153074 1 recorder.go:70] Recording insights-operator/gathers with fingerprint=2910c3786a83b9f6a0fbf0f21d2db2b68b68eb73c1d8524637a6c6d72174f430 I0312 16:08:48.153193 1 diskrecorder.go:70] Writing 109 records to /var/lib/insights-operator/insights-2026-03-12-160848.tar.gz I0312 16:08:48.159857 1 diskrecorder.go:51] Wrote 109 records to disk in 6ms I0312 16:08:48.159892 1 periodic.go:281] Gathering cluster info every 2h0m0s I0312 16:08:48.159909 1 periodic.go:282] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0312 16:09:54.562799 1 diskrecorder.go:223] Found files to send: insights-2026-03-12-160848.tar.gz I0312 16:09:54.562826 1 insightsuploader.go:150] Checking archives to upload periodically every 15m0.616324243s I0312 16:09:54.562836 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0312 16:09:54.577500 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0312 16:09:54.852689 1 requests.go:87] Successfully reported id=2026-03-12T16:09:54Z x-rh-insights-request-id=cc261c767a0c497397a3b48a7e5647fb, wrote=52827 I0312 16:09:54.852745 1 insightsuploader.go:187] Uploaded report successfully in 289.901074ms I0312 16:09:54.852769 1 controller.go:119] Initializing last reported time to 2026-03-12T16:09:54Z I0312 16:09:54.852797 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0312 16:09:54.852810 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0312 16:09:54.852820 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0312 16:09:54.858696 1 controller.go:481] The operator is healthy I0312 16:10:24.580788 1 controller.go:481] The operator is healthy I0312 16:10:57.525487 1 insightsreport.go:137] Pulling report from smart-proxy I0312 16:10:57.525518 1 insightsreport.go:149] Retrieving report I0312 16:10:57.535077 1 requests.go:110] Retrieving report for cluster: 8f8f8919-7f28-431a-90fb-c65df75d00fc I0312 16:10:57.535095 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/8f8f8919-7f28-431a-90fb-c65df75d00fc/reports I0312 16:10:57.538023 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/8f8f8919-7f28-431a-90fb-c65df75d00fc/reports I0312 16:10:57.812613 1 insightsreport.go:184] Report retrieved I0312 16:10:57.838283 1 insightsreport.go:239] Report retrieved correctly I0312 16:12:24.582503 1 controller.go:481] The operator is healthy I0312 16:13:24.555758 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0312 16:13:24.564612 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0312 16:13:24.564630 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0312 16:13:24.569078 1 secretconfigobserver.go:119] support secret does not exist I0312 16:14:24.582691 1 controller.go:481] The operator is healthy I0312 16:16:24.586535 1 controller.go:481] The operator is healthy I0312 16:18:24.552036 1 configmapobserver.go:84] configmaps "insights-config" not found I0312 16:18:24.556732 1 configmapobserver.go:84] configmaps "insights-config" not found I0312 16:18:24.569833 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0312 16:18:24.576881 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0312 16:18:24.576900 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0312 16:18:24.577912 1 controller.go:481] The operator is healthy I0312 16:18:24.581310 1 secretconfigobserver.go:119] support secret does not exist I0312 16:20:24.578418 1 controller.go:481] The operator is healthy I0312 16:22:24.578369 1 controller.go:481] The operator is healthy I0312 16:23:24.582300 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0312 16:23:24.587892 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0312 16:23:24.587913 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0312 16:23:24.592183 1 secretconfigobserver.go:119] support secret does not exist I0312 16:24:24.578430 1 controller.go:481] The operator is healthy I0312 16:25:00.470404 1 insightsuploader.go:146] Nothing to report since 2026-03-12T16:09:54Z I0312 16:26:24.578178 1 controller.go:481] The operator is healthy I0312 16:28:24.551421 1 configmapobserver.go:84] configmaps "insights-config" not found I0312 16:28:24.556039 1 configmapobserver.go:84] configmaps "insights-config" not found I0312 16:28:24.578097 1 controller.go:481] The operator is healthy I0312 16:28:24.592965 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0312 16:28:24.597810 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0312 16:28:24.597828 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0312 16:28:24.601885 1 secretconfigobserver.go:119] support secret does not exist I0312 16:30:24.578319 1 controller.go:481] The operator is healthy I0312 16:32:24.582484 1 controller.go:481] The operator is healthy I0312 16:33:24.602081 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0312 16:33:24.608060 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0312 16:33:24.608078 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0312 16:33:24.612623 1 secretconfigobserver.go:119] support secret does not exist I0312 16:34:24.579046 1 controller.go:481] The operator is healthy I0312 16:36:24.577927 1 controller.go:481] The operator is healthy I0312 16:38:24.551565 1 configmapobserver.go:84] configmaps "insights-config" not found I0312 16:38:24.577413 1 controller.go:481] The operator is healthy I0312 16:38:24.613444 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0312 16:38:24.618389 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0312 16:38:24.618405 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0312 16:38:24.622538 1 secretconfigobserver.go:119] support secret does not exist I0312 16:40:24.578337 1 controller.go:481] The operator is healthy I0312 16:41:33.207445 1 insightsuploader.go:146] Nothing to report since 2026-03-12T16:09:54Z I0312 16:42:24.579114 1 controller.go:481] The operator is healthy I0312 16:43:24.622974 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0312 16:43:24.628041 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0312 16:43:24.628061 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0312 16:43:24.632595 1 secretconfigobserver.go:119] support secret does not exist I0312 16:44:24.579241 1 controller.go:481] The operator is healthy