I0521 19:44:05.125449 1 cmd.go:253] Using service-serving-cert provided certificates I0521 19:44:05.126245 1 observer_polling.go:159] Starting file observer I0521 19:44:05.367310 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0521 19:44:05.367540 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0521 19:44:05.368020 1 secure_serving.go:57] Forcing use of http/1.1 only W0521 19:44:05.368038 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0521 19:44:05.368042 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0521 19:44:05.368046 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0521 19:44:05.368049 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0521 19:44:05.368051 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0521 19:44:05.368054 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0521 19:44:05.368107 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0521 19:44:05.372887 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0521 19:44:05.372911 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:44:05.372868 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0521 19:44:05.372938 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0521 19:44:05.372886 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0521 19:44:05.372958 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:44:05.373040 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0521 19:44:05.373084 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"c6385180-7a7c-42cb-ad45-9204b7adb6bc", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0521 19:44:05.373191 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0521 19:44:05.373671 1 secure_serving.go:213] Serving securely on [::]:8443 I0521 19:44:05.373725 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0521 19:44:05.377967 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0521 19:44:05.377991 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:44:05.378086 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0521 19:44:05.382614 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:44:05.382632 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:44:05.386912 1 secretconfigobserver.go:119] support secret does not exist I0521 19:44:05.390784 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:44:05.394233 1 secretconfigobserver.go:119] support secret does not exist I0521 19:44:05.396660 1 recorder.go:161] Pruning old reports every 5h49m21s, max age is 288h0m0s I0521 19:44:05.400723 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0521 19:44:05.400737 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0521 19:44:05.400742 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0521 19:44:05.400747 1 periodic.go:209] Running clusterconfig gatherer I0521 19:44:05.400751 1 insightsreport.go:296] Starting report retriever I0521 19:44:05.400761 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0521 19:44:05.400785 1 tasks_processing.go:45] number of workers: 64 I0521 19:44:05.400806 1 tasks_processing.go:69] worker 3 listening for tasks. I0521 19:44:05.400817 1 tasks_processing.go:69] worker 8 listening for tasks. I0521 19:44:05.400821 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:44:05.400824 1 tasks_processing.go:69] worker 4 listening for tasks. I0521 19:44:05.400831 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:44:05.400832 1 tasks_processing.go:69] worker 5 listening for tasks. I0521 19:44:05.400836 1 tasks_processing.go:69] worker 6 listening for tasks. I0521 19:44:05.400837 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:44:05.400842 1 tasks_processing.go:69] worker 7 listening for tasks. I0521 19:44:05.400843 1 tasks_processing.go:69] worker 41 listening for tasks. I0521 19:44:05.400850 1 tasks_processing.go:69] worker 29 listening for tasks. I0521 19:44:05.400851 1 tasks_processing.go:69] worker 28 listening for tasks. I0521 19:44:05.400852 1 tasks_processing.go:71] worker 7 working on silenced_alerts task. I0521 19:44:05.400858 1 tasks_processing.go:69] worker 30 listening for tasks. I0521 19:44:05.400862 1 tasks_processing.go:71] worker 28 working on sap_config task. I0521 19:44:05.400867 1 tasks_processing.go:69] worker 43 listening for tasks. I0521 19:44:05.400868 1 tasks_processing.go:71] worker 30 working on openstack_version task. I0521 19:44:05.400871 1 tasks_processing.go:69] worker 58 listening for tasks. I0521 19:44:05.400873 1 tasks_processing.go:69] worker 44 listening for tasks. I0521 19:44:05.400871 1 tasks_processing.go:69] worker 18 listening for tasks. I0521 19:44:05.400881 1 tasks_processing.go:69] worker 13 listening for tasks. I0521 19:44:05.400887 1 tasks_processing.go:69] worker 36 listening for tasks. I0521 19:44:05.400887 1 tasks_processing.go:69] worker 47 listening for tasks. I0521 19:44:05.400862 1 tasks_processing.go:69] worker 57 listening for tasks. I0521 19:44:05.400894 1 tasks_processing.go:69] worker 14 listening for tasks. I0521 19:44:05.400896 1 tasks_processing.go:69] worker 37 listening for tasks. I0521 19:44:05.400892 1 tasks_processing.go:69] worker 60 listening for tasks. I0521 19:44:05.400901 1 tasks_processing.go:69] worker 61 listening for tasks. I0521 19:44:05.400904 1 tasks_processing.go:69] worker 15 listening for tasks. I0521 19:44:05.400901 1 tasks_processing.go:69] worker 12 listening for tasks. I0521 19:44:05.400901 1 tasks_processing.go:69] worker 10 listening for tasks. I0521 19:44:05.400913 1 tasks_processing.go:69] worker 39 listening for tasks. I0521 19:44:05.400916 1 tasks_processing.go:69] worker 50 listening for tasks. I0521 19:44:05.400912 1 tasks_processing.go:69] worker 16 listening for tasks. I0521 19:44:05.400914 1 tasks_processing.go:69] worker 17 listening for tasks. I0521 19:44:05.400922 1 tasks_processing.go:71] worker 3 working on nodenetworkstates task. I0521 19:44:05.400901 1 tasks_processing.go:69] worker 48 listening for tasks. I0521 19:44:05.400925 1 tasks_processing.go:69] worker 40 listening for tasks. I0521 19:44:05.400930 1 tasks_processing.go:71] worker 8 working on openshift_logging task. I0521 19:44:05.400927 1 tasks_processing.go:69] worker 54 listening for tasks. I0521 19:44:05.400932 1 tasks_processing.go:69] worker 51 listening for tasks. I0521 19:44:05.400937 1 tasks_processing.go:69] worker 20 listening for tasks. I0521 19:44:05.400939 1 tasks_processing.go:69] worker 22 listening for tasks. I0521 19:44:05.400940 1 tasks_processing.go:69] worker 31 listening for tasks. I0521 19:44:05.400942 1 tasks_processing.go:69] worker 19 listening for tasks. I0521 19:44:05.400944 1 tasks_processing.go:71] worker 4 working on storage_cluster task. I0521 19:44:05.400951 1 tasks_processing.go:69] worker 25 listening for tasks. I0521 19:44:05.400953 1 tasks_processing.go:71] worker 5 working on image_pruners task. I0521 19:44:05.400950 1 tasks_processing.go:69] worker 34 listening for tasks. I0521 19:44:05.400946 1 tasks_processing.go:69] worker 55 listening for tasks. I0521 19:44:05.400960 1 tasks_processing.go:69] worker 26 listening for tasks. I0521 19:44:05.400958 1 tasks_processing.go:69] worker 24 listening for tasks. I0521 19:44:05.400964 1 tasks_processing.go:71] worker 43 working on overlapping_namespace_uids task. I0521 19:44:05.400971 1 tasks_processing.go:71] worker 37 working on crds task. I0521 19:44:05.400972 1 tasks_processing.go:71] worker 39 working on clusterroles task. I0521 19:44:05.400967 1 tasks_processing.go:69] worker 32 listening for tasks. I0521 19:44:05.400977 1 tasks_processing.go:69] worker 27 listening for tasks. I0521 19:44:05.400981 1 tasks_processing.go:71] worker 1 working on infrastructures task. I0521 19:44:05.400987 1 tasks_processing.go:71] worker 32 working on sap_pods task. I0521 19:44:05.400990 1 tasks_processing.go:71] worker 27 working on certificate_signing_requests task. I0521 19:44:05.400993 1 tasks_processing.go:71] worker 16 working on container_images task. I0521 19:44:05.400995 1 tasks_processing.go:71] worker 58 working on nodenetworkconfigurationpolicies task. I0521 19:44:05.400998 1 tasks_processing.go:71] worker 61 working on proxies task. I0521 19:44:05.400993 1 tasks_processing.go:71] worker 60 working on authentication task. I0521 19:44:05.401066 1 tasks_processing.go:71] worker 44 working on openstack_dataplanenodesets task. I0521 19:44:05.401125 1 tasks_processing.go:71] worker 47 working on operators task. I0521 19:44:05.401133 1 tasks_processing.go:71] worker 17 working on qemu_kubevirt_launcher_logs task. I0521 19:44:05.401159 1 tasks_processing.go:71] worker 57 working on machine_configs task. I0521 19:44:05.401181 1 tasks_processing.go:71] worker 15 working on cost_management_metrics_configs task. I0521 19:44:05.400857 1 tasks_processing.go:71] worker 29 working on aggregated_monitoring_cr_names task. I0521 19:44:05.401216 1 tasks_processing.go:71] worker 36 working on validating_webhook_configurations task. I0521 19:44:05.401251 1 tasks_processing.go:71] worker 18 working on jaegers task. I0521 19:44:05.401315 1 tasks_processing.go:71] worker 10 working on node_logs task. I0521 19:44:05.400895 1 tasks_processing.go:69] worker 11 listening for tasks. I0521 19:44:05.400862 1 tasks_processing.go:69] worker 42 listening for tasks. I0521 19:44:05.400907 1 tasks_processing.go:69] worker 38 listening for tasks. I0521 19:44:05.400907 1 tasks_processing.go:69] worker 49 listening for tasks. I0521 19:44:05.401183 1 tasks_processing.go:71] worker 48 working on nodes task. I0521 19:44:05.401380 1 tasks_processing.go:71] worker 49 working on container_runtime_configs task. I0521 19:44:05.400885 1 tasks_processing.go:69] worker 59 listening for tasks. I0521 19:44:05.401423 1 tasks_processing.go:71] worker 59 working on ingress task. I0521 19:44:05.401426 1 tasks_processing.go:71] worker 14 working on openstack_controlplanes task. I0521 19:44:05.400987 1 tasks_processing.go:71] worker 13 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0521 19:44:05.401452 1 tasks_processing.go:71] worker 19 working on config_maps task. I0521 19:44:05.401454 1 tasks_processing.go:71] worker 22 working on pod_network_connectivity_checks task. I0521 19:44:05.401459 1 tasks_processing.go:71] worker 54 working on machine_autoscalers task. I0521 19:44:05.400911 1 tasks_processing.go:69] worker 9 listening for tasks. I0521 19:44:05.400924 1 tasks_processing.go:69] worker 53 listening for tasks. I0521 19:44:05.400851 1 tasks_processing.go:71] worker 41 working on olm_operators task. I0521 19:44:05.401541 1 tasks_processing.go:71] worker 53 working on machine_healthchecks task. I0521 19:44:05.401551 1 tasks_processing.go:71] worker 38 working on feature_gates task. I0521 19:44:05.400926 1 tasks_processing.go:71] worker 2 working on ingress_certificates task. I0521 19:44:05.401690 1 tasks_processing.go:71] worker 25 working on image task. I0521 19:44:05.400910 1 tasks_processing.go:69] worker 62 listening for tasks. I0521 19:44:05.401840 1 tasks_processing.go:71] worker 62 working on openstack_dataplanedeployments task. I0521 19:44:05.400928 1 tasks_processing.go:69] worker 33 listening for tasks. I0521 19:44:05.400934 1 tasks_processing.go:69] worker 21 listening for tasks. I0521 19:44:05.401453 1 tasks_processing.go:71] worker 20 working on active_alerts task. I0521 19:44:05.400928 1 tasks_processing.go:69] worker 23 listening for tasks. I0521 19:44:05.400856 1 tasks_processing.go:69] worker 56 listening for tasks. I0521 19:44:05.400945 1 tasks_processing.go:69] worker 52 listening for tasks. I0521 19:44:05.400952 1 tasks_processing.go:71] worker 6 working on tsdb_status task. I0521 19:44:05.402021 1 tasks_processing.go:71] worker 52 working on monitoring_persistent_volumes task. I0521 19:44:05.400983 1 tasks_processing.go:71] worker 40 working on mutating_webhook_configurations task. I0521 19:44:05.400982 1 tasks_processing.go:71] worker 12 working on machine_config_pools task. I0521 19:44:05.402150 1 tasks_processing.go:71] worker 21 working on machines task. I0521 19:44:05.401543 1 tasks_processing.go:71] worker 11 working on support_secret task. I0521 19:44:05.402190 1 tasks_processing.go:71] worker 33 working on machine_sets task. I0521 19:44:05.402214 1 tasks_processing.go:71] worker 23 working on image_registries task. I0521 19:44:05.402268 1 tasks_processing.go:71] worker 56 working on cluster_apiserver task. I0521 19:44:05.400851 1 tasks_processing.go:69] worker 35 listening for tasks. I0521 19:44:05.400877 1 tasks_processing.go:69] worker 45 listening for tasks. I0521 19:44:05.401547 1 tasks_processing.go:71] worker 42 working on oauths task. I0521 19:44:05.400882 1 tasks_processing.go:69] worker 46 listening for tasks. I0521 19:44:05.402365 1 tasks_processing.go:71] worker 46 working on ceph_cluster task. I0521 19:44:05.401785 1 tasks_processing.go:71] worker 24 working on service_accounts task. I0521 19:44:05.401790 1 tasks_processing.go:71] worker 34 working on schedulers task. I0521 19:44:05.401626 1 tasks_processing.go:71] worker 51 working on metrics task. I0521 19:44:05.402536 1 tasks_processing.go:71] worker 45 working on operators_pods_and_events task. I0521 19:44:05.402574 1 tasks_processing.go:71] worker 35 working on pdbs task. I0521 19:44:05.400916 1 tasks_processing.go:69] worker 63 listening for tasks. I0521 19:44:05.401803 1 tasks_processing.go:71] worker 55 working on dvo_metrics task. I0521 19:44:05.401793 1 tasks_processing.go:71] worker 26 working on storage_classes task. I0521 19:44:05.400935 1 tasks_processing.go:71] worker 0 working on install_plans task. I0521 19:44:05.401805 1 tasks_processing.go:71] worker 31 working on version task. I0521 19:44:05.400988 1 tasks_processing.go:71] worker 50 working on networks task. I0521 19:44:05.402930 1 tasks_processing.go:71] worker 63 working on openshift_machine_api_events task. I0521 19:44:05.403273 1 tasks_processing.go:71] worker 9 working on lokistack task. I0521 19:44:05.406973 1 tasks_processing.go:71] worker 4 working on sap_datahubs task. I0521 19:44:05.406993 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 6.01596ms to process 0 records I0521 19:44:05.407005 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 6.118168ms to process 0 records I0521 19:44:05.407014 1 tasks_processing.go:74] worker 30 stopped. I0521 19:44:05.407320 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0521 19:44:05.407355 1 controller.go:326] The initial operator extension status is healthy I0521 19:44:05.407365 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:44:05.407369 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:44:05.407372 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0521 19:44:05.407385 1 controller.go:489] The operator is still being initialized I0521 19:44:05.407393 1 controller.go:512] The operator is healthy I0521 19:44:05.407461 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0521 19:44:05.407463 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0521 19:44:05.407593 1 operator.go:288] started I0521 19:44:05.407612 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer E0521 19:44:05.408474 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.241.184:9091: connect: connection refused I0521 19:44:05.408490 1 tasks_processing.go:74] worker 6 stopped. E0521 19:44:05.408480 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.241.184:9091: connect: connection refused E0521 19:44:05.408497 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.241.184:9091: connect: connection refused I0521 19:44:05.408507 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 6.574473ms to process 0 records E0521 19:44:05.408515 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.241.184:9091: connect: connection refused I0521 19:44:05.408532 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 5.986768ms to process 0 records I0521 19:44:05.408543 1 tasks_processing.go:74] worker 51 stopped. E0521 19:44:05.408551 1 gather_silenced_alerts.go:52] Unable to retrieve silenced alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp 172.30.145.253:9094: connect: connection refused I0521 19:44:05.408570 1 tasks_processing.go:74] worker 7 stopped. E0521 19:44:05.408578 1 gather.go:140] gatherer "clusterconfig" function "silenced_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp 172.30.145.253:9094: connect: connection refused E0521 19:44:05.408581 1 gather_active_alerts.go:64] Unable to retrieve most recent alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp 172.30.145.253:9094: connect: connection refused I0521 19:44:05.408602 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 7.705398ms to process 0 records E0521 19:44:05.408624 1 gather.go:140] gatherer "clusterconfig" function "active_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp 172.30.145.253:9094: connect: connection refused I0521 19:44:05.408633 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 6.704138ms to process 0 records I0521 19:44:05.408640 1 tasks_processing.go:74] worker 20 stopped. I0521 19:44:05.417588 1 tasks_processing.go:74] worker 18 stopped. I0521 19:44:05.417604 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 16.32328ms to process 0 records I0521 19:44:05.417614 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 16.120238ms to process 0 records I0521 19:44:05.417621 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 16.524249ms to process 0 records I0521 19:44:05.417623 1 tasks_processing.go:74] worker 54 stopped. I0521 19:44:05.417627 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 16.674721ms to process 0 records I0521 19:44:05.417635 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 16.600262ms to process 0 records I0521 19:44:05.417635 1 tasks_processing.go:74] worker 44 stopped. I0521 19:44:05.417640 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 16.616832ms to process 0 records I0521 19:44:05.417649 1 tasks_processing.go:74] worker 3 stopped. I0521 19:44:05.417645 1 tasks_processing.go:74] worker 32 stopped. I0521 19:44:05.417655 1 tasks_processing.go:74] worker 58 stopped. I0521 19:44:05.417656 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 16.683012ms to process 0 records E0521 19:44:05.417667 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0521 19:44:05.417674 1 tasks_processing.go:74] worker 8 stopped. I0521 19:44:05.417677 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 16.155945ms to process 0 records I0521 19:44:05.417686 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 16.430233ms to process 0 records I0521 19:44:05.417697 1 tasks_processing.go:74] worker 22 stopped. I0521 19:44:05.417699 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 16.75585ms to process 0 records I0521 19:44:05.417704 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 15.461377ms to process 0 records I0521 19:44:05.417705 1 tasks_processing.go:74] worker 28 stopped. I0521 19:44:05.417704 1 tasks_processing.go:74] worker 15 stopped. I0521 19:44:05.417710 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 16.227518ms to process 0 records I0521 19:44:05.417715 1 tasks_processing.go:74] worker 33 stopped. I0521 19:44:05.417716 1 tasks_processing.go:74] worker 14 stopped. I0521 19:44:05.417737 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 15.807797ms to process 0 records I0521 19:44:05.417755 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 16.33612ms to process 0 records I0521 19:44:05.417764 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 15.354412ms to process 0 records I0521 19:44:05.417772 1 tasks_processing.go:74] worker 46 stopped. I0521 19:44:05.417778 1 tasks_processing.go:74] worker 62 stopped. I0521 19:44:05.417783 1 tasks_processing.go:74] worker 49 stopped. I0521 19:44:05.417838 1 tasks_processing.go:74] worker 59 stopped. I0521 19:44:05.418006 1 recorder.go:75] Recording config/ingress with fingerprint=80808da02a92b32edecc5e6bf6fae6afb77c279b6c2fce006d544190899c5fde I0521 19:44:05.418018 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 16.406231ms to process 1 records I0521 19:44:05.418099 1 tasks_processing.go:74] worker 60 stopped. I0521 19:44:05.418166 1 recorder.go:75] Recording config/authentication with fingerprint=b2aa512503f5987f4d20ce30fcf572db395cd48ff9a65f9094c931fdf3c79d0e I0521 19:44:05.418176 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 16.919511ms to process 1 records E0521 19:44:05.418186 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0521 19:44:05.418192 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 16.426982ms to process 0 records I0521 19:44:05.418197 1 tasks_processing.go:74] worker 53 stopped. I0521 19:44:05.418261 1 tasks_processing.go:74] worker 1 stopped. I0521 19:44:05.418626 1 recorder.go:75] Recording config/infrastructure with fingerprint=851d09fb39724268c864ea01ecd77e207de8478c4da636ed57771b7be99c56b5 I0521 19:44:05.418636 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 17.067559ms to process 1 records I0521 19:44:05.418714 1 tasks_processing.go:74] worker 5 stopped. I0521 19:44:05.418808 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=9de19d5436b8e99a4b47c739bf89c60cf1ea1cad17aa3e09a388ceb8a56abcd5 I0521 19:44:05.418817 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 17.12178ms to process 1 records I0521 19:44:05.423777 1 tasks_processing.go:74] worker 61 stopped. I0521 19:44:05.423836 1 recorder.go:75] Recording config/proxy with fingerprint=6411dd38c140da76b2c49d60e69b5fd99e46b27a91ff13d8e35a103c8339b7c5 I0521 19:44:05.423847 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 22.770525ms to process 1 records I0521 19:44:05.429408 1 tasks_processing.go:74] worker 21 stopped. E0521 19:44:05.429425 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0521 19:44:05.429437 1 gather.go:177] gatherer "clusterconfig" function "machines" took 27.246797ms to process 0 records I0521 19:44:05.430000 1 tasks_processing.go:74] worker 42 stopped. I0521 19:44:05.430091 1 gather_logs.go:145] no pods in namespace were found I0521 19:44:05.430240 1 recorder.go:75] Recording config/oauth with fingerprint=d4599d0194729da9a1460e3a70510cf2e3f4fcb6f834b2146062741bc2bec1ea I0521 19:44:05.430253 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 27.674588ms to process 1 records I0521 19:44:05.430259 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 23.11464ms to process 0 records I0521 19:44:05.430263 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 28.96449ms to process 0 records I0521 19:44:05.430268 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 26.866462ms to process 0 records I0521 19:44:05.430273 1 tasks_processing.go:74] worker 9 stopped. I0521 19:44:05.430279 1 tasks_processing.go:74] worker 17 stopped. I0521 19:44:05.430285 1 tasks_processing.go:74] worker 4 stopped. I0521 19:44:05.430467 1 tasks_processing.go:74] worker 11 stopped. E0521 19:44:05.430484 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0521 19:44:05.430495 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 28.269434ms to process 0 records I0521 19:44:05.430654 1 tasks_processing.go:74] worker 56 stopped. I0521 19:44:05.430805 1 recorder.go:75] Recording config/apiserver with fingerprint=69cc677086791c23e138e780819dafa65757b3c56d8b8a6f09db7a2b91be714e I0521 19:44:05.430823 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 28.334593ms to process 1 records I0521 19:44:05.430879 1 tasks_processing.go:74] worker 35 stopped. I0521 19:44:05.430976 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=9962a33b5572c327faf6c63d6a464c65d2c1cb92f3606f2a7e1a8fb119f7693f I0521 19:44:05.431003 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=035459f459e22c78d8bb9506e480a7c1986ca8cb2150032d6e7ffccc681c1415 I0521 19:44:05.431034 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=d89f07c9fc5ceb1ac39b6e8f61531426cf1015d017ca2feb1aa3298d6c521bf6 I0521 19:44:05.431055 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=42eb48b7e6f905fc728da94f30db2f91603dd8d2e131f06277e62f609c6132b1 I0521 19:44:05.431079 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=5469272e424a9f00a59e190aba86b79eb566e4b558fa654bc5f72d1d4fcf7896 I0521 19:44:05.431104 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=0abc7bdfbb7ad02c32908a24820d46e06e84fc72e525e33d7d0355426fff4760 I0521 19:44:05.431126 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=727f892c0fdfd3e902db5ec9ce83f06b47d777f3a4b2976fccc7fcbecb1f3d2a I0521 19:44:05.431149 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=00d297fe73411973cca7ec42b5dbf9a84fba700a96d9e1a30f2f59da57a2a7dc I0521 19:44:05.431172 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=831a7ceef46ab782b03c0499537605ce74c65ea31a0a983084d00a9964ad3a83 I0521 19:44:05.431192 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=f5951a0e1531eff33625f345070c9148f8588920780f108470f316a6dc64e275 I0521 19:44:05.431202 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 28.27367ms to process 10 records I0521 19:44:05.431328 1 tasks_processing.go:74] worker 40 stopped. I0521 19:44:05.431832 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=c3b45b4496bec1cea1f476998994288375e98fa272273a89cf3d0ed3d444d251 I0521 19:44:05.431897 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=ea771990d3d11f55f304cc438ae63a7663a4d3a9e0be062f5aa2310b0f14f21a I0521 19:44:05.431946 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=a4da33b0f7714863f4029a89618fe0aad51b744b1e3847ba98be23c1d48fd17c I0521 19:44:05.431966 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 29.040341ms to process 3 records I0521 19:44:05.432155 1 tasks_processing.go:74] worker 23 stopped. I0521 19:44:05.433006 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=4088805e4712fdeaa2d36818ea6911642d3e53622deeb79c198187f089760476 I0521 19:44:05.433164 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 29.383818ms to process 1 records I0521 19:44:05.433264 1 tasks_processing.go:74] worker 38 stopped. I0521 19:44:05.433589 1 recorder.go:75] Recording config/featuregate with fingerprint=ea6335774ebd697d41ba0b7485279af0faca9898ce761c0d56ed769f5f4ef3bb I0521 19:44:05.433626 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 30.566074ms to process 1 records I0521 19:44:05.433640 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 30.104113ms to process 0 records I0521 19:44:05.433652 1 tasks_processing.go:74] worker 12 stopped. I0521 19:44:05.434751 1 tasks_processing.go:74] worker 52 stopped. I0521 19:44:05.434769 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 32.712522ms to process 0 records I0521 19:44:05.434800 1 tasks_processing.go:74] worker 25 stopped. I0521 19:44:05.434946 1 recorder.go:75] Recording config/image with fingerprint=8bd3fdc37bb13d1b7701c10b0984da0c76efcadc04dbd176096925510a88b463 I0521 19:44:05.434966 1 gather.go:177] gatherer "clusterconfig" function "image" took 33.050352ms to process 1 records I0521 19:44:05.435034 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=dda5e9c12e85f8f6bc818d484b119233ca8e2a0852d72b601164c974261cdc54 I0521 19:44:05.435048 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 32.390404ms to process 1 records I0521 19:44:05.435086 1 tasks_processing.go:74] worker 34 stopped. I0521 19:44:05.435153 1 tasks_processing.go:74] worker 50 stopped. I0521 19:44:05.435206 1 recorder.go:75] Recording config/network with fingerprint=bc893a545cbba1e4031232ca1e03161f96566e049fc075b11a4e1a93103bec02 I0521 19:44:05.435218 1 gather.go:177] gatherer "clusterconfig" function "networks" took 31.945624ms to process 1 records I0521 19:44:05.435755 1 tasks_processing.go:74] worker 27 stopped. I0521 19:44:05.435785 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 34.742158ms to process 0 records I0521 19:44:05.436408 1 tasks_processing.go:74] worker 26 stopped. I0521 19:44:05.436525 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=3f34815ee837a8b8cacb4687887d325bdf1dc0c751dfe9b698c4b35f1a62309d I0521 19:44:05.436552 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=54479d9a7b00bf38ed13ac8bfc72d00c176fc4ce8643cc6820ac073ac401d1f9 I0521 19:44:05.436565 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 33.500254ms to process 2 records I0521 19:44:05.436572 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 35.15955ms to process 0 records I0521 19:44:05.436577 1 tasks_processing.go:74] worker 10 stopped. I0521 19:44:05.436820 1 tasks_processing.go:74] worker 63 stopped. I0521 19:44:05.436840 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 33.686768ms to process 0 records I0521 19:44:05.436985 1 tasks_processing.go:74] worker 48 stopped. I0521 19:44:05.437491 1 recorder.go:75] Recording config/node/ip-10-0-0-127.ec2.internal with fingerprint=9bb52724590190bcd5ab3bd99c6a1f1ba952251f35be195f074c61a9f6a9200a I0521 19:44:05.437604 1 recorder.go:75] Recording config/node/ip-10-0-1-62.ec2.internal with fingerprint=c656a1d71e205cecd6be49a60a2be0732649d2ed80da6ebd14485ba21d729a0a I0521 19:44:05.437766 1 recorder.go:75] Recording config/node/ip-10-0-2-95.ec2.internal with fingerprint=ae20f2ad6c74a4fa667170ee4cedf58d0d43bece7d3d95faa9e5e2fc8df23c2f I0521 19:44:05.437783 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 35.599896ms to process 3 records I0521 19:44:05.437883 1 tasks_processing.go:74] worker 39 stopped. I0521 19:44:05.438367 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=6c0430b24115c00057ce7057c60a8b9c7c42eb595f76a4f0aff973698d638d92 W0521 19:44:05.438506 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:44:05.438628 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=4dcd884c3d3769ac7360209b93513e68ff1ecec75e263335e0b87fdea61471aa I0521 19:44:05.438643 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 36.293766ms to process 2 records I0521 19:44:05.438673 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0521 19:44:05.438688 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 36.503555ms to process 1 records I0521 19:44:05.438699 1 tasks_processing.go:74] worker 43 stopped. I0521 19:44:05.439296 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:44:05.439367 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0521 19:44:05.439379 1 controller.go:212] Source scaController *sca.Controller is not ready I0521 19:44:05.439384 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0521 19:44:05.439389 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:44:05.439464 1 controller.go:489] The operator is still being initialized I0521 19:44:05.439477 1 controller.go:512] The operator is healthy I0521 19:44:05.440443 1 tasks_processing.go:74] worker 36 stopped. I0521 19:44:05.440919 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=e5783060a8b2cd0eefdaa81eb42361df8f555a40775d043e62522fe3b315b233 I0521 19:44:05.440987 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=c8bf489f2a8009b2e0cc736684527b0182d69b0c3aec2828a79f1047a503cc41 I0521 19:44:05.441073 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=fa369a94921010b605ef0857250a941a8a912ad14a333d97f18a977b85798f1d I0521 19:44:05.441108 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=9e51b10201a42bd143937a8db1ffc713d9a8e9a9117d599d4178ad4820883abf I0521 19:44:05.441165 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=4f9600d8374f9b2728045fe8f47ff4e25f7f273436d7f0fc1389a2a1205e948a I0521 19:44:05.441199 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=c31b9fb936e4b380214c75886fc44c55d3f1049b2aee162d937d87ada7653aed I0521 19:44:05.441233 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=7fd5baa8270553cd4a149eb1a80f2c96279817d390b64aa7a1e471a3425b8fd8 I0521 19:44:05.441266 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=8b157050ab045eb63a7462b51edca433bd80552bc80d98dfccd5a5ea2530d75d I0521 19:44:05.441299 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=66bd47d610d5b4f52593818a023067ab587b4018baef075f897e4cd7ae13f8bd I0521 19:44:05.441364 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=f501b0f7074c834f3aec0d616e83c407e183973e59797ae756fe95348297d481 I0521 19:44:05.441400 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=97cf42a1de6caa01c80a7546b697d4fd41e2040964c0187a048b2a059864d38f I0521 19:44:05.441434 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=00ddbd27eb5a8067a0c08d834a8b0771b06c1f951fd5e016c375beb144a9b091 I0521 19:44:05.441467 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=937e9d0503055098bcdacedae938bedc8ffeaaada17e378f172ac643bd65ca9d I0521 19:44:05.441479 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 39.211005ms to process 13 records I0521 19:44:05.442457 1 tasks_processing.go:74] worker 45 stopped. I0521 19:44:05.442480 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 39.909049ms to process 0 records I0521 19:44:05.446261 1 tasks_processing.go:74] worker 37 stopped. I0521 19:44:05.446801 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=210dd2e2585c541be7e91c4fb64368ca873935f8915192b6de187d537038329f I0521 19:44:05.447019 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=75e32d05ae43969e8a956d13289d3e8341398e86991cffdc0be25505212da048 I0521 19:44:05.447029 1 gather.go:177] gatherer "clusterconfig" function "crds" took 45.277512ms to process 2 records I0521 19:44:05.447637 1 tasks_processing.go:74] worker 31 stopped. I0521 19:44:05.448072 1 recorder.go:75] Recording config/version with fingerprint=c33214f07500ca5aae7310859be7738cf157e87cd385ce46ba65384f8cbe9f18 I0521 19:44:05.448095 1 recorder.go:75] Recording config/id with fingerprint=7dcc7250af2a34c126d1f4e84c0342e5884fe44b2856deee1319bd204b78c039 I0521 19:44:05.448106 1 gather.go:177] gatherer "clusterconfig" function "version" took 44.647466ms to process 2 records I0521 19:44:05.448187 1 tasks_processing.go:74] worker 41 stopped. I0521 19:44:05.448239 1 recorder.go:75] Recording config/olm_operators with fingerprint=2f797def0c962623df54775527be1d8b6b76f6c8862f307498241e35a30af8c2 I0521 19:44:05.448254 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 46.687623ms to process 1 records I0521 19:44:05.450634 1 tasks_processing.go:74] worker 57 stopped. I0521 19:44:05.450675 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0521 19:44:05.450689 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 49.44936ms to process 1 records I0521 19:44:05.451624 1 tasks_processing.go:74] worker 29 stopped. I0521 19:44:05.451655 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 50.424627ms to process 0 records I0521 19:44:05.454193 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0521 19:44:05.457942 1 tasks_processing.go:74] worker 19 stopped. E0521 19:44:05.457957 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0521 19:44:05.457962 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0521 19:44:05.457967 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0521 19:44:05.457976 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0521 19:44:05.458001 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0521 19:44:05.458011 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0521 19:44:05.458015 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0521 19:44:05.458024 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0521 19:44:05.458085 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0521 19:44:05.458102 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0521 19:44:05.458110 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 56.474288ms to process 7 records I0521 19:44:05.461361 1 tasks_processing.go:74] worker 16 stopped. I0521 19:44:05.462642 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-n664m with fingerprint=699426e407f73754d0ebf948e82982ab2a34703fdb6f25bca539bff7e96768f8 I0521 19:44:05.462727 1 recorder.go:75] Recording config/running_containers with fingerprint=1597d7d9bef948b6fe909f0a70f8744fbd4a6ff160319a3b627756fd8b21e3f8 I0521 19:44:05.462738 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 60.352958ms to process 2 records E0521 19:44:05.462749 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0521 19:44:05.462754 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2qegoo3vnj953sjchii9ofsr5vsfv02i-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2qegoo3vnj953sjchii9ofsr5vsfv02i-primary-cert-bundle-secret" not found I0521 19:44:05.462763 1 tasks_processing.go:74] worker 2 stopped. I0521 19:44:05.462790 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=ef272bc3aa65d34a1d4883c3c4a93cbf9928ee23d5496909aec7b5ce06b3f527 I0521 19:44:05.462801 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 59.932404ms to process 1 records I0521 19:44:05.462810 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 61.193025ms to process 0 records I0521 19:44:05.462817 1 tasks_processing.go:74] worker 13 stopped. I0521 19:44:05.472966 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:44:05.473048 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0521 19:44:05.473131 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:44:05.478160 1 base_controller.go:82] Caches are synced for ConfigController I0521 19:44:05.478175 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0521 19:44:05.482975 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:44:05.507980 1 base_controller.go:82] Caches are synced for LoggingSyncer I0521 19:44:05.507999 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... W0521 19:44:06.439294 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0521 19:44:07.438959 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:44:07.865076 1 tasks_processing.go:74] worker 47 stopped. I0521 19:44:07.865195 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=3a293012a496c86bd211e5329a780ce282ad027bc41a8c5ecca3ba76faaeda0c I0521 19:44:07.865244 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0521 19:44:07.865267 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=dc8aed552c738fe9ee8652087ea7ba8aedf8af0f7f05ad732e6a91b754ef931e I0521 19:44:07.865277 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0521 19:44:07.865300 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=63b400297fb969ea4cfc540c1161987cced2f8f5dc9668816f9085a01ead50c8 I0521 19:44:07.865315 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0521 19:44:07.865357 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=6edd469dd58d38ec4722fd31b90f388ab52e0425f66c674002bfc589cb108918 I0521 19:44:07.865385 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=e73dd7df57955d90f3f02c8030642a10459e0fad736a3fe87211bd129a9844db I0521 19:44:07.865409 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=b8e250a840b28e63fa612280f9c12fa4bcb17ffe5a61636302b1b33b54e7de00 I0521 19:44:07.865441 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=87434b0d454977058459a2a14bf987f3dbe367e39cbae7c8b450bd73317d28f4 I0521 19:44:07.865700 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=162c1d78373d00b5f2f457fa0fdf7ec1918d43c26518b232a07e4a8956f094c9 I0521 19:44:07.865719 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=5624ad5967503d37c6b4d263c26a1309915f4bbfbf2b7ce6bec3424bc702a8ec I0521 19:44:07.865730 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0521 19:44:07.865745 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=5e136bac633da702c1fc3a263f21ef5553cf9d01c2501de79a4903e3b4f10c3a I0521 19:44:07.865756 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0521 19:44:07.865772 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=b6059b544b268cdc4df3f359ac26b4b43f4c47df0df382ad6cef36890ab36083 I0521 19:44:07.865780 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0521 19:44:07.865799 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=b780c2eca4506d48900d9c23de6f05ec543a6ae2034fc8e697ce31f7b1b36085 I0521 19:44:07.865809 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0521 19:44:07.865829 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=b7004972518a4480812e39676a87abc5c906eb3a2642d8ce135d27fd3296ea25 I0521 19:44:07.865930 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=649e4adf763fcb20813076df70699a6770260519a8fdd99564819ea4ac5a3866 I0521 19:44:07.865939 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0521 19:44:07.865948 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0521 19:44:07.865970 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0521 19:44:07.865998 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=3a4cd1feaf8c8701792459ad9224edde8153307013a5627d9ca5c8134f7eb08f I0521 19:44:07.866022 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=9f8dd4967176484dba9f663c57e185ca5b97fa074f527daffcadd97446a6f259 I0521 19:44:07.866031 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0521 19:44:07.866058 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=7495c0b3c11567c3e499681ccd323e8f0a06cbb4ac33ffd4ecdb29254ceadd5e I0521 19:44:07.866068 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0521 19:44:07.866090 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=d0eef32f25d5b8fa961409dd7ef37f96d778b429a7d5ab764411b627e6b3c929 I0521 19:44:07.866100 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0521 19:44:07.866113 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=f13930a50631b36c7703f826ad5a87fde9cd7f795b963237d506aeb662c133a7 I0521 19:44:07.866129 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=6e8a8ad87bda5334b8bef50f4cd877e8b0bac9d646d21fafc11a98f59172c671 I0521 19:44:07.866145 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=1f167ff095668dadf55bafac4934bc3e86a7f964a1082508996e407a55fccd76 I0521 19:44:07.866167 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=3cf2f119feb1efa26307c275cbdcc89b29acc84deb94c1585f89a9fb36fa3ab7 I0521 19:44:07.866176 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0521 19:44:07.866199 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=da2dcbf4233e677fe7c4752b7f2f3417cbe5d9599f1c282dfdb08135c5c76f08 I0521 19:44:07.866218 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0521 19:44:07.866228 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0521 19:44:07.866234 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.463924495s to process 39 records W0521 19:44:08.438357 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0521 19:44:09.438398 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0521 19:44:10.434229 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:44:10.434254 1 tasks_processing.go:74] worker 55 stopped. E0521 19:44:10.434276 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:44:10.434295 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0521 19:44:10.434306 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.031527871s to process 1 records I0521 19:44:18.441641 1 tasks_processing.go:74] worker 0 stopped. I0521 19:44:18.441705 1 recorder.go:75] Recording config/installplans with fingerprint=95dfd5f33a9a46199239158a0ec0183b629818eca24ae41f4d706d7a3d604aec I0521 19:44:18.441722 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 13.038702851s to process 1 records I0521 19:44:19.208533 1 tasks_processing.go:74] worker 24 stopped. I0521 19:44:19.208810 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=ae29e7659fb4cd0f04bcc181913d3f8b69c20a612f3ed6bf2f2479314a8cc58f I0521 19:44:19.208827 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.806086095s to process 1 records E0521 19:44:19.208890 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.808s with: function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"silenced_alerts\" failed with an error, function \"active_alerts\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0521 19:44:19.209992 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "tsdb_status" failed with an error, function "metrics" failed with an error, function "silenced_alerts" failed with an error, function "active_alerts" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0521 19:44:19.210009 1 periodic.go:209] Running workloads gatherer I0521 19:44:19.210026 1 tasks_processing.go:45] number of workers: 2 I0521 19:44:19.210033 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:44:19.210038 1 tasks_processing.go:71] worker 1 working on workload_info task. I0521 19:44:19.210063 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:44:19.210081 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0521 19:44:19.219745 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-127.ec2.internal... I0521 19:44:19.219764 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-95.ec2.internal... I0521 19:44:19.219752 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-62.ec2.internal... I0521 19:44:19.233043 1 tasks_processing.go:74] worker 0 stopped. I0521 19:44:19.233057 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 22.933226ms to process 0 records I0521 19:44:19.435938 1 controllerstatus.go:89] name=scaController healthy=false reason=Unauthorized message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: OCM API https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates returned HTTP 401: {"code":"ACCT-MGMT-15","href":"/api/accounts_mgmt/v1/errors/15","id":"15","kind":"Error","operation_id":"f274c763-c54d-48fb-b55b-90f8b52de2ff","reason":"Unable to get payload details from JWT token: Unable to retrieve JWT token from request context"} I0521 19:44:19.805803 1 controllerstatus.go:89] name=clusterTransferController healthy=false reason=Unauthorized message=failed to pull cluster transfer: OCM API https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27701b401c-0f18-4913-ad4c-066b87957d03%27+and+status+is+%27accepted%27 returned HTTP 401: {"code":"ACCT-MGMT-15","href":"/api/accounts_mgmt/v1/errors/15","id":"15","kind":"Error","operation_id":"71f31b62-6e53-4c97-8e21-7613a92c0e67","reason":"Unable to get payload details from JWT token: Unable to retrieve JWT token from request context"} I0521 19:44:24.006685 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 5s I0521 19:44:24.033302 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 25s for image data I0521 19:44:24.041428 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (9ms) I0521 19:44:24.050122 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (9ms) I0521 19:44:24.059402 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (9ms) I0521 19:44:24.067947 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (9ms) I0521 19:44:24.076623 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (9ms) I0521 19:44:24.085764 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (9ms) I0521 19:44:24.095473 1 gather_workloads_info.go:387] No image sha256:90bdf9e678fe18e35401e34e1cb0912d822e9cb32b9e41cc545870540e9a87f2 (10ms) I0521 19:44:24.104213 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (9ms) I0521 19:44:24.112777 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (9ms) I0521 19:44:24.121805 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (9ms) I0521 19:44:24.142317 1 gather_workloads_info.go:387] No image sha256:b99613494e9d6a4a8207323872d63cc02b830fc80a8bf972e62c64381aa4cd17 (21ms) I0521 19:44:24.242269 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (100ms) I0521 19:44:24.341656 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (99ms) I0521 19:44:24.442090 1 gather_workloads_info.go:387] No image sha256:513b1166e376feca472fdd5174f6abf0abf02925d6a21b7b4d89a2ff2fc67bed (100ms) I0521 19:44:24.541711 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (100ms) I0521 19:44:24.641888 1 gather_workloads_info.go:387] No image sha256:7bd3e88d6d3f878a89203df590b684c5a77bf08df4717451d3197ab74683a80e (100ms) I0521 19:44:24.741693 1 gather_workloads_info.go:387] No image sha256:6e5df2aab9ce759e8f957341a11c087c233307e3c8f8760d21270efc19ccd3ce (100ms) I0521 19:44:24.842359 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (101ms) I0521 19:44:24.942407 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0521 19:44:25.044610 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (102ms) I0521 19:44:25.142322 1 gather_workloads_info.go:387] No image sha256:1477f3287cc75352305274f248f781c1b7c2a8696f4f7bc450c45a4a5f5b10ea (98ms) I0521 19:44:25.242230 1 gather_workloads_info.go:387] No image sha256:101993f6e65d6fda8b90c583b718b720809182961f69bc777295f47e6df4d7ce (100ms) I0521 19:44:25.341608 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (99ms) I0521 19:44:25.441680 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (100ms) I0521 19:44:25.545915 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (104ms) I0521 19:44:25.642231 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (96ms) I0521 19:44:25.741725 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (99ms) I0521 19:44:25.841526 1 gather_workloads_info.go:387] No image sha256:1e65f9f5490fbc930d2dce96f03ddcb28990b839d7dc011ef5de5bec6c5095bf (100ms) I0521 19:44:25.941288 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (100ms) I0521 19:44:26.041209 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (100ms) I0521 19:44:26.141461 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (100ms) I0521 19:44:26.242512 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (101ms) I0521 19:44:26.341873 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (99ms) I0521 19:44:26.444690 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (103ms) I0521 19:44:26.541806 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (97ms) I0521 19:44:26.642371 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (101ms) I0521 19:44:26.742606 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (100ms) I0521 19:44:26.842004 1 gather_workloads_info.go:387] No image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (99ms) I0521 19:44:26.942273 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (100ms) I0521 19:44:27.042267 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0521 19:44:27.141155 1 gather_workloads_info.go:387] No image sha256:0092c6ff139f8698c91349d507eb3182bc5d6c4ae040f672e347ff504accdea2 (99ms) I0521 19:44:27.244118 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (103ms) I0521 19:44:27.341641 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (98ms) I0521 19:44:27.441445 1 gather_workloads_info.go:387] No image sha256:0af73ad7a47c9ff53269ae2bc5163bf9e389085220c34e4bd42c92c4e7a85f6d (100ms) I0521 19:44:27.542354 1 gather_workloads_info.go:387] No image sha256:74713675d580a47a56b628469c4c05c41e114e0b935960d718ff0cf7e72f613d (101ms) I0521 19:44:27.643868 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (101ms) I0521 19:44:27.742214 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (98ms) I0521 19:44:27.843850 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (102ms) I0521 19:44:27.843882 1 tasks_processing.go:74] worker 1 stopped. I0521 19:44:27.844267 1 recorder.go:75] Recording config/workload_info with fingerprint=9391b92355c62ca93ead8bab69090b9afeab075fb200bf2d84e82e592ba4b3be I0521 19:44:27.844286 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.633836583s to process 1 records I0521 19:44:27.844305 1 periodic.go:256] Periodic gather workloads completed in 8.634s I0521 19:44:27.844313 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0521 19:44:27.844318 1 periodic.go:209] Running conditional gatherer I0521 19:44:27.850401 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0521 19:44:42.822851 1 conditional_gatherer.go:282] Received HTTP status code 401, trying again in 30s I0521 19:45:12.829866 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0521 19:45:27.868058 1 conditional_gatherer.go:282] Received HTTP status code 401, trying again in 1m0s I0521 19:45:27.868084 1 conditional_gatherer.go:107] received HTTP 401 Unauthorized from https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules. Using the default built-in configuration I0521 19:45:27.868440 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0521 19:45:28.891291 1 conditional_gatherer.go:374] alert "PrometheusKubernetesListWatchFailures" has state "pending" I0521 19:45:28.891305 1 conditional_gatherer.go:374] alert "KubePodNotReady" has state "pending" I0521 19:45:28.891308 1 conditional_gatherer.go:374] alert "KubeStatefulSetReplicasMismatch" has state "pending" I0521 19:45:28.891311 1 conditional_gatherer.go:374] alert "KubeDaemonSetRolloutStuck" has state "pending" I0521 19:45:28.891314 1 conditional_gatherer.go:374] alert "KubeDaemonSetRolloutStuck" has state "pending" I0521 19:45:28.891316 1 conditional_gatherer.go:374] alert "KubeContainerWaiting" has state "pending" I0521 19:45:28.891318 1 conditional_gatherer.go:374] alert "KubeContainerWaiting" has state "pending" I0521 19:45:28.891321 1 conditional_gatherer.go:374] alert "KubeContainerWaiting" has state "pending" I0521 19:45:28.891323 1 conditional_gatherer.go:374] alert "KubePdbNotEnoughHealthyPods" has state "pending" I0521 19:45:28.891325 1 conditional_gatherer.go:374] alert "Watchdog" has state "firing" I0521 19:45:28.891328 1 conditional_gatherer.go:374] alert "ClusterMonitoringOperatorReconciliationErrors" has state "pending" I0521 19:45:28.891331 1 conditional_gatherer.go:374] alert "AlertmanagerReceiversNotConfigured" has state "pending" I0521 19:45:28.891349 1 conditional_gatherer.go:374] alert "KubePodNotScheduled" has state "pending" I0521 19:45:28.891416 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0521 19:45:28.899619 1 conditional_gatherer.go:392] cluster version is '4.20.8' I0521 19:45:28.899659 1 tasks_processing.go:45] number of workers: 3 I0521 19:45:28.899673 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:45:28.899680 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0521 19:45:28.899685 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:45:28.899698 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0521 19:45:28.899699 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:45:28.899710 1 tasks_processing.go:74] worker 1 stopped. I0521 19:45:28.899704 1 tasks_processing.go:71] worker 0 working on conditional_gatherer_rules task. I0521 19:45:28.899794 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0521 19:45:28.899818 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.411µs to process 1 records I0521 19:45:28.899888 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=5e6e9deebe7018085c2e386bec070d88cace102ade375a07031a50ddcdc08e82 I0521 19:45:28.899898 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.168µs to process 1 records I0521 19:45:28.899900 1 tasks_processing.go:74] worker 0 stopped. I0521 19:45:28.899978 1 tasks_processing.go:74] worker 2 stopped. I0521 19:45:28.899993 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 291.632µs to process 0 records I0521 19:45:28.900015 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=HttpStatus401 message=received HTTP 401 Unauthorized from https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules. Using the default built-in configuration I0521 19:45:28.900043 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0521 19:45:28.924800 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=47ff16b8410e1fb5a22e0fb688995ad6710d46150c9c677138358b7593ef4426 I0521 19:45:28.924926 1 diskrecorder.go:70] Writing 108 records to /var/lib/insights-operator/insights-2026-05-21-194528.tar.gz I0521 19:45:28.931595 1 diskrecorder.go:51] Wrote 108 records to disk in 6ms I0521 19:45:28.931630 1 periodic.go:278] Gathering cluster info every 2h0m0s I0521 19:45:28.931652 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0521 19:45:35.401122 1 diskrecorder.go:223] Found files to send: insights-2026-05-21-194528.tar.gz I0521 19:45:35.401154 1 insightsuploader.go:150] Checking archives to upload periodically every 15m31.244739015s I0521 19:45:35.401163 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0521 19:45:35.409885 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0521 19:45:50.597903 1 insightsuploader.go:169] Unable to upload report after 15.19s: gateway server reported unexpected error code: 504 (request=bbc6e69687874083a747317a37dcfc8f): {"errors":[{"meta":{"response_by":"gateway"},"status":504,"detail":"UHC services authentication failed"}]} I0521 19:45:50.597929 1 controllerstatus.go:80] name=insightsuploader healthy=false reason=UploadFailed message=Unable to report: gateway server reported unexpected error code: 504 (request=bbc6e69687874083a747317a37dcfc8f): {"errors":[{"meta":{"response_by":"gateway"},"status":504,"detail":"UHC services authentication failed"}]} I0521 19:46:05.413472 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:46:05.413508 1 controller.go:512] The operator is healthy I0521 19:48:05.413724 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:48:05.413763 1 controller.go:512] The operator is healthy I0521 19:49:05.394564 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 19:49:05.399577 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:49:05.399601 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 19:49:05.403371 1 secretconfigobserver.go:119] support secret does not exist I0521 19:50:05.413707 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:50:05.413740 1 controller.go:512] The operator is healthy I0521 19:52:05.423640 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:52:05.423675 1 controller.go:512] The operator is healthy I0521 19:54:05.393740 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:54:05.397749 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:54:05.403461 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 19:54:05.407389 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:54:05.407408 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 19:54:05.411658 1 secretconfigobserver.go:119] support secret does not exist I0521 19:54:05.412269 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:54:05.412304 1 controller.go:512] The operator is healthy I0521 19:56:05.413637 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:56:05.413678 1 controller.go:512] The operator is healthy I0521 19:58:05.413311 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:58:05.413371 1 controller.go:512] The operator is healthy