I0504 09:24:48.212739 1 cmd.go:253] Using service-serving-cert provided certificates I0504 09:24:48.213157 1 observer_polling.go:159] Starting file observer I0504 09:24:48.494359 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0504 09:24:48.494567 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0504 09:24:48.495109 1 secure_serving.go:57] Forcing use of http/1.1 only W0504 09:24:48.495149 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0504 09:24:48.495154 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0504 09:24:48.495159 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0504 09:24:48.495162 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. I0504 09:24:48.495157 1 simple_featuregate_reader.go:171] Starting feature-gate-detector W0504 09:24:48.495164 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0504 09:24:48.495181 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0504 09:24:48.498306 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0504 09:24:48.498325 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"60c3718f-810e-4e88-919b-8ea362709e33", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0504 09:24:48.500144 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0504 09:24:48.500164 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0504 09:24:48.500164 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0504 09:24:48.500178 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0504 09:24:48.500186 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0504 09:24:48.500175 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0504 09:24:48.500449 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0504 09:24:48.500685 1 secure_serving.go:213] Serving securely on [::]:8443 I0504 09:24:48.500713 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0504 09:24:48.508524 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0504 09:24:48.508546 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0504 09:24:48.508653 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0504 09:24:48.513384 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 09:24:48.513400 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0504 09:24:48.518546 1 secretconfigobserver.go:119] support secret does not exist I0504 09:24:48.522682 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 09:24:48.526356 1 secretconfigobserver.go:119] support secret does not exist I0504 09:24:48.529656 1 recorder.go:176] Pruning old reports every 5h58m23s, max age is 288h0m0s I0504 09:24:48.538658 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0504 09:24:48.538663 1 periodic.go:216] Running clusterconfig gatherer I0504 09:24:48.538675 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0504 09:24:48.538704 1 tasks_processing.go:45] number of workers: 64 I0504 09:24:48.538710 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0504 09:24:48.538714 1 insightsreport.go:296] Starting report retriever I0504 09:24:48.538718 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0504 09:24:48.538727 1 tasks_processing.go:69] worker 9 listening for tasks. I0504 09:24:48.538733 1 tasks_processing.go:69] worker 8 listening for tasks. I0504 09:24:48.538737 1 tasks_processing.go:69] worker 7 listening for tasks. I0504 09:24:48.538741 1 tasks_processing.go:69] worker 25 listening for tasks. I0504 09:24:48.538735 1 tasks_processing.go:69] worker 5 listening for tasks. I0504 09:24:48.538746 1 tasks_processing.go:69] worker 3 listening for tasks. I0504 09:24:48.538749 1 tasks_processing.go:71] worker 5 working on machine_config_pools task. I0504 09:24:48.538747 1 tasks_processing.go:69] worker 10 listening for tasks. I0504 09:24:48.538754 1 tasks_processing.go:69] worker 26 listening for tasks. I0504 09:24:48.538755 1 tasks_processing.go:69] worker 2 listening for tasks. I0504 09:24:48.538748 1 tasks_processing.go:71] worker 25 working on sap_pods task. I0504 09:24:48.538765 1 tasks_processing.go:69] worker 28 listening for tasks. I0504 09:24:48.538767 1 tasks_processing.go:69] worker 13 listening for tasks. I0504 09:24:48.538770 1 tasks_processing.go:69] worker 4 listening for tasks. I0504 09:24:48.538774 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 09:24:48.538769 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 09:24:48.538772 1 tasks_processing.go:69] worker 29 listening for tasks. I0504 09:24:48.538781 1 tasks_processing.go:69] worker 31 listening for tasks. I0504 09:24:48.538761 1 tasks_processing.go:69] worker 27 listening for tasks. I0504 09:24:48.538776 1 tasks_processing.go:69] worker 30 listening for tasks. I0504 09:24:48.538779 1 tasks_processing.go:69] worker 14 listening for tasks. I0504 09:24:48.538786 1 tasks_processing.go:69] worker 6 listening for tasks. I0504 09:24:48.538793 1 tasks_processing.go:69] worker 15 listening for tasks. I0504 09:24:48.538795 1 tasks_processing.go:69] worker 21 listening for tasks. I0504 09:24:48.538796 1 tasks_processing.go:69] worker 18 listening for tasks. I0504 09:24:48.538759 1 tasks_processing.go:69] worker 12 listening for tasks. I0504 09:24:48.538805 1 tasks_processing.go:69] worker 40 listening for tasks. I0504 09:24:48.538810 1 tasks_processing.go:69] worker 35 listening for tasks. I0504 09:24:48.538783 1 tasks_processing.go:69] worker 19 listening for tasks. I0504 09:24:48.538814 1 tasks_processing.go:69] worker 36 listening for tasks. I0504 09:24:48.538788 1 tasks_processing.go:69] worker 16 listening for tasks. I0504 09:24:48.538788 1 tasks_processing.go:69] worker 20 listening for tasks. I0504 09:24:48.538789 1 tasks_processing.go:69] worker 17 listening for tasks. I0504 09:24:48.538797 1 tasks_processing.go:69] worker 50 listening for tasks. I0504 09:24:48.538830 1 tasks_processing.go:69] worker 55 listening for tasks. I0504 09:24:48.538922 1 tasks_processing.go:71] worker 8 working on machines task. I0504 09:24:48.538958 1 tasks_processing.go:69] worker 45 listening for tasks. I0504 09:24:48.538751 1 tasks_processing.go:69] worker 11 listening for tasks. I0504 09:24:48.538957 1 tasks_processing.go:71] worker 7 working on config_maps task. I0504 09:24:48.539425 1 tasks_processing.go:71] worker 45 working on support_secret task. I0504 09:24:48.540049 1 tasks_processing.go:69] worker 38 listening for tasks. I0504 09:24:48.540094 1 tasks_processing.go:69] worker 46 listening for tasks. I0504 09:24:48.540103 1 tasks_processing.go:69] worker 52 listening for tasks. I0504 09:24:48.540109 1 tasks_processing.go:69] worker 37 listening for tasks. I0504 09:24:48.540163 1 tasks_processing.go:69] worker 22 listening for tasks. I0504 09:24:48.540168 1 tasks_processing.go:69] worker 56 listening for tasks. I0504 09:24:48.540109 1 tasks_processing.go:69] worker 32 listening for tasks. I0504 09:24:48.540195 1 tasks_processing.go:69] worker 59 listening for tasks. I0504 09:24:48.540214 1 tasks_processing.go:71] worker 38 working on sap_config task. I0504 09:24:48.540247 1 tasks_processing.go:69] worker 23 listening for tasks. I0504 09:24:48.540157 1 tasks_processing.go:69] worker 51 listening for tasks. I0504 09:24:48.540249 1 tasks_processing.go:69] worker 58 listening for tasks. I0504 09:24:48.540249 1 tasks_processing.go:71] worker 9 working on sap_datahubs task. I0504 09:24:48.540372 1 tasks_processing.go:71] worker 17 working on image_registries task. I0504 09:24:48.540141 1 tasks_processing.go:69] worker 24 listening for tasks. I0504 09:24:48.540436 1 tasks_processing.go:71] worker 6 working on openstack_version task. I0504 09:24:48.540455 1 tasks_processing.go:71] worker 20 working on version task. I0504 09:24:48.540453 1 tasks_processing.go:71] worker 36 working on validating_webhook_configurations task. I0504 09:24:48.540253 1 tasks_processing.go:69] worker 57 listening for tasks. I0504 09:24:48.540478 1 tasks_processing.go:71] worker 50 working on authentication task. I0504 09:24:48.540504 1 tasks_processing.go:71] worker 21 working on container_images task. I0504 09:24:48.538945 1 tasks_processing.go:69] worker 49 listening for tasks. I0504 09:24:48.540271 1 tasks_processing.go:71] worker 35 working on machine_autoscalers task. I0504 09:24:48.540628 1 tasks_processing.go:69] worker 62 listening for tasks. I0504 09:24:48.540645 1 tasks_processing.go:69] worker 60 listening for tasks. I0504 09:24:48.540651 1 tasks_processing.go:69] worker 41 listening for tasks. I0504 09:24:48.540601 1 tasks_processing.go:71] worker 18 working on machine_sets task. I0504 09:24:48.539409 1 tasks_processing.go:69] worker 42 listening for tasks. I0504 09:24:48.539415 1 tasks_processing.go:69] worker 43 listening for tasks. I0504 09:24:48.540761 1 tasks_processing.go:71] worker 29 working on node_features task. I0504 09:24:48.540650 1 tasks_processing.go:69] worker 61 listening for tasks. I0504 09:24:48.540860 1 tasks_processing.go:71] worker 55 working on ingress_certificates task. I0504 09:24:48.540267 1 tasks_processing.go:69] worker 48 listening for tasks. I0504 09:24:48.540271 1 tasks_processing.go:69] worker 53 listening for tasks. I0504 09:24:48.540348 1 tasks_processing.go:71] worker 15 working on cost_management_metrics_configs task. I0504 09:24:48.539421 1 tasks_processing.go:71] worker 3 working on clusterroles task. I0504 09:24:48.541158 1 tasks_processing.go:71] worker 53 working on mutating_webhook_configurations task. I0504 09:24:48.539430 1 tasks_processing.go:71] worker 26 working on cluster_apiserver task. I0504 09:24:48.540423 1 tasks_processing.go:71] worker 14 working on jaegers task. I0504 09:24:48.540344 1 tasks_processing.go:71] worker 40 working on proxies task. I0504 09:24:48.539432 1 tasks_processing.go:71] worker 2 working on crds task. I0504 09:24:48.540348 1 tasks_processing.go:71] worker 19 working on operators task. I0504 09:24:48.541522 1 tasks_processing.go:69] worker 54 listening for tasks. I0504 09:24:48.540863 1 tasks_processing.go:69] worker 44 listening for tasks. I0504 09:24:48.541054 1 tasks_processing.go:71] worker 11 working on feature_gates task. I0504 09:24:48.541753 1 tasks_processing.go:71] worker 23 working on infrastructures task. I0504 09:24:48.540263 1 tasks_processing.go:71] worker 16 working on container_runtime_configs task. I0504 09:24:48.541912 1 tasks_processing.go:69] worker 63 listening for tasks. I0504 09:24:48.539426 1 tasks_processing.go:71] worker 10 working on openstack_controlplanes task. I0504 09:24:48.541923 1 tasks_processing.go:71] worker 63 working on qemu_kubevirt_launcher_logs task. I0504 09:24:48.540265 1 tasks_processing.go:69] worker 34 listening for tasks. I0504 09:24:48.539452 1 tasks_processing.go:71] worker 12 working on openshift_machine_api_events task. I0504 09:24:48.541972 1 tasks_processing.go:71] worker 34 working on install_plans task. I0504 09:24:48.541743 1 tasks_processing.go:71] worker 44 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0504 09:24:48.542152 1 tasks_processing.go:71] worker 4 working on image_pruners task. I0504 09:24:48.541713 1 tasks_processing.go:71] worker 54 working on oauths task. I0504 09:24:48.542168 1 tasks_processing.go:71] worker 37 working on ceph_cluster task. I0504 09:24:48.541724 1 tasks_processing.go:71] worker 22 working on openstack_dataplanenodesets task. I0504 09:24:48.539459 1 tasks_processing.go:71] worker 27 working on pod_network_connectivity_checks task. I0504 09:24:48.541724 1 tasks_processing.go:71] worker 13 working on pdbs task. I0504 09:24:48.539465 1 tasks_processing.go:69] worker 33 listening for tasks. I0504 09:24:48.541741 1 tasks_processing.go:71] worker 24 working on silenced_alerts task. I0504 09:24:48.539436 1 tasks_processing.go:71] worker 28 working on ingress task. I0504 09:24:48.541746 1 tasks_processing.go:71] worker 57 working on nodenetworkconfigurationpolicies task. I0504 09:24:48.541743 1 tasks_processing.go:71] worker 42 working on nodenetworkstates task. I0504 09:24:48.541748 1 tasks_processing.go:71] worker 56 working on schedulers task. I0504 09:24:48.541754 1 tasks_processing.go:71] worker 59 working on monitoring_persistent_volumes task. I0504 09:24:48.541758 1 tasks_processing.go:71] worker 32 working on machine_configs task. I0504 09:24:48.541754 1 tasks_processing.go:71] worker 62 working on overlapping_namespace_uids task. I0504 09:24:48.543084 1 tasks_processing.go:71] worker 52 working on metrics task. I0504 09:24:48.543085 1 tasks_processing.go:71] worker 46 working on tsdb_status task. I0504 09:24:48.541767 1 tasks_processing.go:71] worker 61 working on networks task. I0504 09:24:48.543238 1 tasks_processing.go:71] worker 38 working on dvo_metrics task. I0504 09:24:48.543260 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 3.012395ms to process 0 records I0504 09:24:48.543317 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 4.496489ms to process 0 records I0504 09:24:48.541767 1 tasks_processing.go:71] worker 51 working on active_alerts task. I0504 09:24:48.543385 1 tasks_processing.go:71] worker 5 working on openstack_dataplanedeployments task. I0504 09:24:48.541769 1 tasks_processing.go:71] worker 48 working on nodes task. I0504 09:24:48.539462 1 tasks_processing.go:71] worker 30 working on machine_healthchecks task. I0504 09:24:48.543529 1 tasks_processing.go:71] worker 41 working on aggregated_monitoring_cr_names task. I0504 09:24:48.543548 1 tasks_processing.go:71] worker 60 working on openshift_logging task. I0504 09:24:48.539441 1 tasks_processing.go:71] worker 1 working on olm_operators task. I0504 09:24:48.539439 1 tasks_processing.go:71] worker 0 working on service_accounts task. I0504 09:24:48.543697 1 tasks_processing.go:71] worker 49 working on certificate_signing_requests task. I0504 09:24:48.541726 1 tasks_processing.go:71] worker 58 working on image task. I0504 09:24:48.543397 1 tasks_processing.go:69] worker 47 listening for tasks. I0504 09:24:48.543796 1 tasks_processing.go:71] worker 47 working on storage_classes task. I0504 09:24:48.543554 1 tasks_processing.go:71] worker 43 working on node_logs task. I0504 09:24:48.539456 1 tasks_processing.go:71] worker 31 working on lokistack task. I0504 09:24:48.543884 1 tasks_processing.go:71] worker 33 working on operators_pods_and_events task. I0504 09:24:48.539455 1 tasks_processing.go:69] worker 39 listening for tasks. I0504 09:24:48.543929 1 tasks_processing.go:71] worker 39 working on storage_cluster task. E0504 09:24:48.544859 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0504 09:24:48.544874 1 tasks_processing.go:74] worker 29 stopped. I0504 09:24:48.544885 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 4.103606ms to process 0 records I0504 09:24:48.545048 1 tasks_processing.go:74] worker 8 stopped. E0504 09:24:48.545065 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0504 09:24:48.545078 1 gather.go:177] gatherer "clusterconfig" function "machines" took 6.079939ms to process 0 records I0504 09:24:48.545510 1 tasks_processing.go:74] worker 6 stopped. I0504 09:24:48.545523 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 5.060088ms to process 0 records I0504 09:24:48.545606 1 tasks_processing.go:74] worker 35 stopped. I0504 09:24:48.545613 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 4.984667ms to process 0 records I0504 09:24:48.545798 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0504 09:24:48.545818 1 controller.go:368] The initial operator extension status is healthy I0504 09:24:48.545830 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0504 09:24:48.545838 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0504 09:24:48.545843 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0504 09:24:48.545861 1 controller.go:531] The operator is still being initialized I0504 09:24:48.545875 1 controller.go:554] The operator is healthy I0504 09:24:48.545965 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0504 09:24:48.546058 1 operator.go:328] started I0504 09:24:48.546059 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0504 09:24:48.546089 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer E0504 09:24:48.547623 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.57.155:9091: connect: connection refused I0504 09:24:48.547642 1 tasks_processing.go:74] worker 46 stopped. E0504 09:24:48.547649 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.57.155:9091: connect: connection refused I0504 09:24:48.547657 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 4.536962ms to process 0 records E0504 09:24:48.547665 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.57.155:9091: connect: connection refused I0504 09:24:48.547683 1 tasks_processing.go:74] worker 52 stopped. E0504 09:24:48.547690 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.57.155:9091: connect: connection refused I0504 09:24:48.547702 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 4.587167ms to process 0 records I0504 09:24:48.553876 1 tasks_processing.go:74] worker 15 stopped. I0504 09:24:48.553890 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 12.942371ms to process 0 records I0504 09:24:48.553905 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 12.655493ms to process 0 records I0504 09:24:48.553915 1 tasks_processing.go:74] worker 14 stopped. I0504 09:24:48.553927 1 tasks_processing.go:74] worker 18 stopped. I0504 09:24:48.553937 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 13.261735ms to process 0 records I0504 09:24:48.553944 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 13.563005ms to process 0 records I0504 09:24:48.553948 1 tasks_processing.go:74] worker 9 stopped. I0504 09:24:48.553951 1 tasks_processing.go:74] worker 16 stopped. I0504 09:24:48.553954 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 12.101286ms to process 0 records I0504 09:24:48.554095 1 tasks_processing.go:74] worker 40 stopped. I0504 09:24:48.554191 1 recorder.go:75] Recording config/proxy with fingerprint=5d29290f254d612fb1120aba0cfac20e9f760540bcd9f84d6689a3699b859c3e I0504 09:24:48.554204 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 12.825339ms to process 1 records I0504 09:24:48.554229 1 tasks_processing.go:74] worker 26 stopped. I0504 09:24:48.554359 1 recorder.go:75] Recording config/apiserver with fingerprint=7684d613c6ba65930af275a5e448df72db402bbac17727d87808343cdd762f8b I0504 09:24:48.554371 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 12.996915ms to process 1 records I0504 09:24:48.559704 1 tasks_processing.go:74] worker 23 stopped. I0504 09:24:48.560956 1 recorder.go:75] Recording config/infrastructure with fingerprint=37f3195d82860523fe84234791678a92bba994513be27a399d219f43d2f90555 I0504 09:24:48.561030 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 17.897014ms to process 1 records I0504 09:24:48.561530 1 recorder.go:75] Recording config/authentication with fingerprint=fc994bf48b72699bc6ce21acae4bcef556bf22eea81d690fc87dfc43f1871c9f I0504 09:24:48.561581 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 19.560522ms to process 1 records I0504 09:24:48.561701 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=37eb0666cd26761fc91c3d9630fefd9a02e345a14891d08dfabafad1c77833e3 I0504 09:24:48.561723 1 tasks_processing.go:74] worker 50 stopped. I0504 09:24:48.561729 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 17.431132ms to process 1 records I0504 09:24:48.561775 1 tasks_processing.go:74] worker 56 stopped. I0504 09:24:48.561872 1 tasks_processing.go:74] worker 17 stopped. I0504 09:24:48.562775 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=d8656372dac18c9086b439f0f3736b3bcbbba21a12223284575c26916ce85698 I0504 09:24:48.562795 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 20.271957ms to process 1 records I0504 09:24:48.562886 1 tasks_processing.go:74] worker 54 stopped. I0504 09:24:48.563000 1 recorder.go:75] Recording config/oauth with fingerprint=003641286de03e7937305d5e5544e30d19c16b994018ca05e750dd8d5002fb5c I0504 09:24:48.563014 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 19.709512ms to process 1 records I0504 09:24:48.563029 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 19.470315ms to process 0 records I0504 09:24:48.563042 1 tasks_processing.go:74] worker 57 stopped. I0504 09:24:48.564988 1 tasks_processing.go:74] worker 10 stopped. I0504 09:24:48.565000 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 23.056064ms to process 0 records I0504 09:24:48.565009 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 22.424335ms to process 0 records I0504 09:24:48.565015 1 tasks_processing.go:74] worker 42 stopped. I0504 09:24:48.569057 1 tasks_processing.go:74] worker 39 stopped. I0504 09:24:48.569070 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 25.120779ms to process 0 records I0504 09:24:48.569079 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 26.654467ms to process 0 records I0504 09:24:48.569086 1 tasks_processing.go:74] worker 22 stopped. I0504 09:24:48.569089 1 tasks_processing.go:74] worker 12 stopped. I0504 09:24:48.569101 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 27.080429ms to process 0 records I0504 09:24:48.569199 1 tasks_processing.go:74] worker 31 stopped. I0504 09:24:48.569212 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 25.36478ms to process 0 records I0504 09:24:48.569263 1 tasks_processing.go:74] worker 4 stopped. I0504 09:24:48.569346 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=9c1e4aa16751a460739b01b7ca88841dc0a51893f3d7fef0a3fa1674b0e65d2c I0504 09:24:48.569358 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 27.045272ms to process 1 records I0504 09:24:48.572000 1 tasks_processing.go:74] worker 27 stopped. E0504 09:24:48.572013 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0504 09:24:48.572020 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 29.57548ms to process 0 records I0504 09:24:48.572241 1 tasks_processing.go:74] worker 53 stopped. I0504 09:24:48.572386 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=1d80eaa263a3f4cd1661b5cfcd99b105a1adbaf600a017076feb7545c437b35c I0504 09:24:48.572417 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=1f4eb7d6068aa984e4f9cb2bf10bb71c651bbb73862710af11f4efd290be6796 I0504 09:24:48.572442 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=6d7a0bef76eed30dcf456962c3c0b7c7a5d82ba2e785e3033278b6372c2a0d1e I0504 09:24:48.572450 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 31.069176ms to process 3 records I0504 09:24:48.572459 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 30.001256ms to process 0 records I0504 09:24:48.572464 1 tasks_processing.go:74] worker 37 stopped. I0504 09:24:48.576091 1 tasks_processing.go:74] worker 2 stopped. I0504 09:24:48.576817 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=8e714d6d73f0bd79b8f825032950045b34a76d5c3ca63a2321d3f5f9d8aceb3a I0504 09:24:48.577137 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=c71480dda428b3521c0b39b01e53356bd44d6201b44d39492ba7621ac14dcf2c I0504 09:24:48.577153 1 gather.go:177] gatherer "clusterconfig" function "crds" took 34.678045ms to process 2 records I0504 09:24:48.577168 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 32.842691ms to process 0 records E0504 09:24:48.577175 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0504 09:24:48.577183 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 32.827513ms to process 0 records I0504 09:24:48.577183 1 tasks_processing.go:74] worker 5 stopped. I0504 09:24:48.577191 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 32.836981ms to process 0 records I0504 09:24:48.577195 1 tasks_processing.go:74] worker 30 stopped. I0504 09:24:48.577201 1 tasks_processing.go:74] worker 60 stopped. I0504 09:24:48.577269 1 tasks_processing.go:74] worker 11 stopped. I0504 09:24:48.577329 1 recorder.go:75] Recording config/featuregate with fingerprint=6e9c0a0b6a2c03124531220fd9f6e0f703e4a51ae00e119e38e32587aebdc653 I0504 09:24:48.577344 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 34.936634ms to process 1 records I0504 09:24:48.583894 1 tasks_processing.go:74] worker 25 stopped. I0504 09:24:48.583903 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 45.121345ms to process 0 records I0504 09:24:48.584374 1 tasks_processing.go:74] worker 28 stopped. I0504 09:24:48.584486 1 recorder.go:75] Recording config/ingress with fingerprint=c55b61b7116768cd8b92ed907fd35350a4052ab61dd2a78c317b96879625b9b6 I0504 09:24:48.584500 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 41.900571ms to process 1 records I0504 09:24:48.584555 1 gather_logs.go:145] no pods in namespace were found I0504 09:24:48.584572 1 tasks_processing.go:74] worker 63 stopped. I0504 09:24:48.584585 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 42.639139ms to process 0 records I0504 09:24:48.584798 1 tasks_processing.go:74] worker 61 stopped. I0504 09:24:48.584932 1 recorder.go:75] Recording config/network with fingerprint=e222f2928448e0b0732b37de0744e33f636251c2b315444178b63b3b4ce0c1f6 I0504 09:24:48.584944 1 gather.go:177] gatherer "clusterconfig" function "networks" took 41.562311ms to process 1 records I0504 09:24:48.586147 1 tasks_processing.go:74] worker 43 stopped. I0504 09:24:48.586214 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 42.336859ms to process 0 records I0504 09:24:48.586390 1 tasks_processing.go:74] worker 59 stopped. I0504 09:24:48.586403 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 43.734808ms to process 0 records I0504 09:24:48.592277 1 tasks_processing.go:74] worker 20 stopped. I0504 09:24:48.592395 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0504 09:24:48.592413 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0504 09:24:48.592419 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0504 09:24:48.592425 1 controller.go:254] Source scaController *sca.Controller is not ready I0504 09:24:48.592429 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0504 09:24:48.592451 1 controller.go:531] The operator is still being initialized I0504 09:24:48.592463 1 controller.go:554] The operator is healthy I0504 09:24:48.592559 1 recorder.go:75] Recording config/version with fingerprint=8a9d73567b9c17e5967eac28d2662eb61286df5a071076dadc1c4f6d9ca3686d I0504 09:24:48.592591 1 recorder.go:75] Recording config/id with fingerprint=194777a5c98c43eef3439f669691f85520f565b0cd090ec07392cf1703df6052 I0504 09:24:48.592606 1 gather.go:177] gatherer "clusterconfig" function "version" took 51.781507ms to process 2 records I0504 09:24:48.594608 1 tasks_processing.go:74] worker 47 stopped. I0504 09:24:48.594692 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=7303baee3f54b1eccd77b8c92dcc50ec9a66e1d339708862c5f244678b5a5859 I0504 09:24:48.594709 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=a931e203647940cc5d090fcd72a47290958a269e6f24bf16493a483f769fb41a I0504 09:24:48.594715 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 50.803316ms to process 2 records E0504 09:24:48.594722 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0504 09:24:48.594727 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 55.28404ms to process 0 records I0504 09:24:48.594732 1 tasks_processing.go:74] worker 45 stopped. I0504 09:24:48.594802 1 tasks_processing.go:74] worker 58 stopped. I0504 09:24:48.594924 1 recorder.go:75] Recording config/image with fingerprint=178c2680daffaebd300f25527a18958de10b3a74c063449874c2a96a6654b3f6 I0504 09:24:48.594944 1 gather.go:177] gatherer "clusterconfig" function "image" took 51.088856ms to process 1 records I0504 09:24:48.595586 1 tasks_processing.go:74] worker 48 stopped. I0504 09:24:48.595865 1 recorder.go:75] Recording config/node/ip-10-0-0-177.ec2.internal with fingerprint=944f35c01fee154023b23b377076d9d49af1787aebfb0d492635cdb66d20cf0f I0504 09:24:48.595924 1 recorder.go:75] Recording config/node/ip-10-0-1-152.ec2.internal with fingerprint=96564a1037000ff56c6e62048010b0aeb7b0f3edce5c261383c38222a47bced5 I0504 09:24:48.595972 1 recorder.go:75] Recording config/node/ip-10-0-2-89.ec2.internal with fingerprint=16f5729503402a7e661e7c6d8a7f570232ae6dc109cd72067b90402ac1c01735 I0504 09:24:48.595981 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 52.108064ms to process 3 records W0504 09:24:48.596992 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 09:24:48.597892 1 tasks_processing.go:74] worker 13 stopped. I0504 09:24:48.598244 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=9956ab277730b5219fe10e6025cc6ab5ec7fe75d7558173c01d6ccc7777df685 I0504 09:24:48.598281 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=86785fcc47812f700d7fb2a936b193bdd741d8ab7b0d41659bb5545cd36e91b6 I0504 09:24:48.598318 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=f58670e4c9e34c48fd8279dd44a81b75c05449987d46c1c4bdfc3717d6d0612e I0504 09:24:48.598345 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=0627e9b9a8e7e1f18dcc52563e255cb76e2add69accd84e4febb2ed339a0537f I0504 09:24:48.598380 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=5a29159049a3c1af1ed4f08a853979c056a7baaffca87d4cac5353e1773155c8 I0504 09:24:48.598557 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=4cac41e4345702322a414bf52fa094fd1f0b20b94f6164fa199fed6cfc466c21 I0504 09:24:48.598595 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=25f94bb99ab02adc83b51f2ff425971b5df383cd3340bb126c26c1e941139b28 I0504 09:24:48.598638 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=b22b4565e2d22758fa34b8b0d6e96d08b97254c99efed29e61ff60166d155dba I0504 09:24:48.598664 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=ed0ea2fccc657b2cb23da26e7bfd7ef3279008871ca4b71874795504a76a132c I0504 09:24:48.598704 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=842f12a886105c8a962fa6858a328a850a4b24cdaedd6f6dbb0c091f2efdd011 I0504 09:24:48.598744 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=97fe1d08614d9fd3cc9e1e44f2d16389db1d76b01c2a35a281cde0c70c712ad9 I0504 09:24:48.598761 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 55.452976ms to process 11 records I0504 09:24:48.598774 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 54.509425ms to process 0 records E0504 09:24:48.598785 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0504 09:24:48.598818 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0504 09:24:48.598831 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 55.997893ms to process 1 records I0504 09:24:48.598895 1 tasks_processing.go:74] worker 33 stopped. I0504 09:24:48.598896 1 tasks_processing.go:74] worker 32 stopped. I0504 09:24:48.598983 1 tasks_processing.go:74] worker 36 stopped. I0504 09:24:48.598989 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=180b3175dee577183800de0a6d89621eef4ee0e9147aa1fd9954314896e19cf1 I0504 09:24:48.599058 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=3cf3e55f976da34ee30a7525452809ce5c3b4728a7ef53a8f57fe0af014c44fd I0504 09:24:48.599199 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=0050ed6d05026302b711f367b22c04b1adb8776ddd707ff3ee6c60318aafd611 I0504 09:24:48.599256 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=72b4f968dbf132398ae44a1cca406cfcde78a617e4f0a8ec2fbd61b3a82d9040 I0504 09:24:48.599301 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=5dfc52d744e29d2d57c766102f0091e2309478a61799dfa41382c3cf28fa2df9 I0504 09:24:48.599348 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=9c955899e28d77fcd2e5cd3cb3070a3086bd5560c34f30af44b3ab5714d499b4 I0504 09:24:48.599393 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=68cea1d7c0f83192872366a117543441b724870833f980b019896aa0df3086fc I0504 09:24:48.599432 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=16ef1152e6ec4468195969a22bd67c93b3e6ebfb0ea6447af7c120f5634b4626 I0504 09:24:48.599477 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=a3392a7d2a5d2a60f0f566336380912d315625dc9fb501e10092c59aa196a025 I0504 09:24:48.599531 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=b493fdda7f74ce251a49def4fa53ee0324fc2e904c3e598bef37a0276d87552d I0504 09:24:48.599577 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=33f89c8c15cb65fe7d4d48898cdec9855fe55f31456d3742e7e2017117eac31c I0504 09:24:48.599614 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=74ff6effe2b5253c2e12b54291eea696af8799a7040b33c8b88888bf3b25945f I0504 09:24:48.599658 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=feacfcae8186964fed982d58ea5e63fce30db9ef61c83b2e6b40bda8d0dcf27e I0504 09:24:48.599669 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 58.289929ms to process 13 records I0504 09:24:48.600092 1 tasks_processing.go:74] worker 62 stopped. I0504 09:24:48.600195 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0504 09:24:48.600217 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 57.013495ms to process 1 records I0504 09:24:48.600263 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0504 09:24:48.600275 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0504 09:24:48.600321 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0504 09:24:48.603366 1 tasks_processing.go:74] worker 49 stopped. I0504 09:24:48.603389 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 59.650465ms to process 0 records I0504 09:24:48.605959 1 tasks_processing.go:74] worker 3 stopped. I0504 09:24:48.606034 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0504 09:24:48.606203 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=ff637267b44c788a41577af90d13433d74069b937997fd17bac0f770a9dc4fa2 I0504 09:24:48.606299 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=71e3b5c58d2eaebfc3a0849fc4931daa3db0a51463fbb96175045bd045ffa1c4 I0504 09:24:48.606309 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 64.858239ms to process 2 records I0504 09:24:48.607822 1 recorder.go:75] Recording config/olm_operators with fingerprint=5ab980979c336630276d4a9224f1c1af37bff63dc59f19dae0c44ed1de397150 I0504 09:24:48.607677 1 tasks_processing.go:74] worker 1 stopped. I0504 09:24:48.607893 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 63.983817ms to process 1 records I0504 09:24:48.608469 1 tasks_processing.go:74] worker 21 stopped. I0504 09:24:48.608877 1 base_controller.go:82] Caches are synced for ConfigController I0504 09:24:48.608893 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0504 09:24:48.610109 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-66dbbb6468-xd5h5 with fingerprint=71f258175122505a25c08c356ae861e381431448bbdef3cc8b7851e13c9f6dd4 I0504 09:24:48.610225 1 recorder.go:75] Recording config/running_containers with fingerprint=9c643dfbff2fe51bb359ecdb7948e3142ca937cd09a4f13458402cb3d41ebf39 I0504 09:24:48.610235 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 67.951947ms to process 2 records I0504 09:24:48.611680 1 tasks_processing.go:74] worker 41 stopped. I0504 09:24:48.611695 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 68.089696ms to process 0 records I0504 09:24:48.616028 1 tasks_processing.go:74] worker 55 stopped. E0504 09:24:48.616042 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0504 09:24:48.616048 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q30v0mp9l76e0dgvrf8upih6itusg2g-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q30v0mp9l76e0dgvrf8upih6itusg2g-primary-cert-bundle-secret" not found I0504 09:24:48.616088 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=aa2d48f8ed250301a1bb8d45a817b9ed436c3c132029e16ec7bbeb59d71c8137 I0504 09:24:48.616099 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 75.144324ms to process 1 records I0504 09:24:48.616111 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 73.893538ms to process 0 records I0504 09:24:48.616142 1 tasks_processing.go:74] worker 44 stopped. I0504 09:24:48.620354 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 09:24:48.620858 1 tasks_processing.go:74] worker 7 stopped. E0504 09:24:48.620871 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0504 09:24:48.620876 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0504 09:24:48.620879 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0504 09:24:48.620887 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0504 09:24:48.620938 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0504 09:24:48.620945 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0504 09:24:48.620949 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0504 09:24:48.620960 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0504 09:24:48.621010 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0504 09:24:48.621017 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0504 09:24:48.621022 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 81.418392ms to process 7 records I0504 09:24:48.646925 1 base_controller.go:82] Caches are synced for LoggingSyncer I0504 09:24:48.646936 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0504 09:24:48.662025 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0504 09:24:48.666136 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0504 09:24:48.732850 1 cluster_transfer.go:102] no available accepted cluster transfer I0504 09:24:48.732868 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0504 09:24:48.797767 1 sca.go:266] etc-pki-entitlement secret successfully created I0504 09:24:48.797781 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0504 09:24:49.588877 1 tasks_processing.go:74] worker 24 stopped. I0504 09:24:49.588902 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0504 09:24:49.588915 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.046416254s to process 1 records I0504 09:24:49.588996 1 tasks_processing.go:74] worker 51 stopped. I0504 09:24:49.589031 1 recorder.go:75] Recording config/alerts with fingerprint=5f18df36282dade13e0c26b1878154a7de3172b17cfee9b4c84dada2fe40ffd4 I0504 09:24:49.589046 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.045652974s to process 1 records W0504 09:24:49.596995 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0504 09:24:50.597267 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 09:24:51.024843 1 tasks_processing.go:74] worker 19 stopped. I0504 09:24:51.024959 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=d5ece6b07a7b2e69ea2b69ae6a4fed968ed7cf7298e1108494db7142a9527d6c I0504 09:24:51.025006 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0504 09:24:51.025040 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=60ffb97b0b9c952a2378a9d08e6948e1a3dad34296692fa99baa75fa2f6f908b I0504 09:24:51.025050 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0504 09:24:51.025070 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=94e00e09486dd4504456ee6aefc12078cf20474455c4ed2f44dae5ce5d8e0ba4 I0504 09:24:51.025085 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0504 09:24:51.025105 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=e0b35eb8ea97c54e2af7e87c18e02025f86fa81948425975dac1f5d63080f4f8 I0504 09:24:51.025154 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=5bcfba1a8abdbae5c7b48714d0e3fb4dc4c8eac3beba05bfe2fd03ff37021e6e I0504 09:24:51.025178 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=28452572ee6b10ba1d8edcb39c82f5c6fc8923600e7635397851345cb4b5c229 I0504 09:24:51.025207 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=83af66696db653882b328a9692658e1f7c3b7acf23c2fd6f8f78c9cacd468847 I0504 09:24:51.025499 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=d830965eb7d888986b1695252e9f6c58321e6feea0a27b5d28bdd2b2a99a838a I0504 09:24:51.025520 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=07df2c3bff1615433e34d953a3f5396956d824d87b8b089e5912366fa2bda616 I0504 09:24:51.025529 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0504 09:24:51.025545 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=8cb921e4b58b253ad7fb12d2a2d9aa45fffa790ab773587225ff819d91347eba I0504 09:24:51.025557 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0504 09:24:51.025574 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=9ba33367a500d20729273afce3f5f6cedb188f3e44081a4f8972d7a96973708a I0504 09:24:51.025584 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0504 09:24:51.025602 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=0f9bd34f2253f474c8f6524bf177e0922080e6bf55f9771521b033cea675e028 I0504 09:24:51.025612 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0504 09:24:51.025631 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=a6cbd0a46897a5ff2e67bc9ef1294bdc47e4482f0f316e1ab59abcc0201e6f96 I0504 09:24:51.025734 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=3e01a9902534cfcdd0c594d407616a98ef46706d55c7a009b262fcb0302bf1da I0504 09:24:51.025743 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0504 09:24:51.025749 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0504 09:24:51.025769 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0504 09:24:51.025793 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=6c348c0a4553519e51a15496fb111688730b85013f7f5d11ba45422ab6031876 I0504 09:24:51.025816 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=e552acbf61f7722b3d3bd3c3c5000889083f07ae0ddd72c626d439f02e20bd5b I0504 09:24:51.025825 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0504 09:24:51.025840 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=8335741b4ef052ec2239d93aa5e83dd7f9014fbb97bf731ac6fa00bbf17de888 I0504 09:24:51.025849 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0504 09:24:51.025868 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=11effd32ffb69d8ca1213ab118325cf395cd5ce4ef5baec2f21357d838b62d19 I0504 09:24:51.025877 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0504 09:24:51.025890 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=113253d7adce90606f72b958c06c62b7e37b5759d0f44620051b5630ba380ff0 I0504 09:24:51.025903 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=fc305d8710e70456032d95ec5b0967c14641073d7e403156c7162426b1a49d06 I0504 09:24:51.025919 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=47b3470943b0ef19f4f843990c87f1ddf70681246bff8520039a3aec26dbc857 I0504 09:24:51.025938 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=1f2158ddbcad1711d193e9cb7d2c5c97d86b78cbcd26cb364aec5a24689d91d5 I0504 09:24:51.025947 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0504 09:24:51.025976 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=1f1943e0a05f8c7ef5405e608976eec21bb944b85d910714f27ca98b1a12ca92 I0504 09:24:51.025992 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0504 09:24:51.026001 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0504 09:24:51.026008 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.483330004s to process 39 records W0504 09:24:51.596955 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0504 09:24:52.597281 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0504 09:24:53.597991 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0504 09:24:53.598013 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0504 09:24:53.598029 1 tasks_processing.go:74] worker 38 stopped. E0504 09:24:53.598039 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0504 09:24:53.598050 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0504 09:24:53.598059 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.054777097s to process 1 records I0504 09:25:01.398718 1 tasks_processing.go:74] worker 34 stopped. I0504 09:25:01.398764 1 recorder.go:75] Recording config/installplans with fingerprint=95dfd5f33a9a46199239158a0ec0183b629818eca24ae41f4d706d7a3d604aec I0504 09:25:01.398777 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.856720962s to process 1 records I0504 09:25:02.149937 1 tasks_processing.go:74] worker 0 stopped. I0504 09:25:02.150246 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=796bd9f20e825fd32af435d21e60c10c3ec4910fa1211c5a6338f5b81c3f8071 I0504 09:25:02.150265 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.606230361s to process 1 records E0504 09:25:02.150329 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.611s with: function \"machines\" failed with an error, function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"machine_configs\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error" I0504 09:25:02.151441 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machines" failed with an error, function "tsdb_status" failed with an error, function "metrics" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "machine_configs" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error I0504 09:25:02.151457 1 periodic.go:216] Running workloads gatherer I0504 09:25:02.151474 1 tasks_processing.go:45] number of workers: 2 I0504 09:25:02.151487 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 09:25:02.151492 1 tasks_processing.go:71] worker 1 working on workload_info task. I0504 09:25:02.151504 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 09:25:02.151520 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0504 09:25:02.159816 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-89.ec2.internal... I0504 09:25:02.159821 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-177.ec2.internal... I0504 09:25:02.159838 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-152.ec2.internal... I0504 09:25:02.181152 1 tasks_processing.go:74] worker 0 stopped. I0504 09:25:02.181167 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 29.618502ms to process 0 records I0504 09:25:06.990625 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 5s I0504 09:25:07.017324 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0504 09:25:07.025636 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (10ms) I0504 09:25:07.034968 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (9ms) I0504 09:25:07.043307 1 gather_workloads_info.go:387] No image sha256:52db05084f2a2e7d7bf4040e4865df88ec4074efbbe59a78d861be2c5ab2aa40 (8ms) I0504 09:25:07.051987 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (9ms) I0504 09:25:07.059982 1 gather_workloads_info.go:387] No image sha256:e1ba458cf6f0b3606c90880da72db8ab99cd11040bae84baebe3ff2e0d1ea075 (8ms) I0504 09:25:07.068737 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (9ms) I0504 09:25:07.076612 1 gather_workloads_info.go:387] No image sha256:5d6a2b8d2508c7da71192661e7c0dcf1335f48b6336a20189abffbe79eeff962 (8ms) I0504 09:25:07.084855 1 gather_workloads_info.go:398] Found image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (8ms) I0504 09:25:07.093320 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (8ms) I0504 09:25:07.101518 1 gather_workloads_info.go:387] No image sha256:0acfe677ed58031f8381cb8a4f2d8917ec4cfa95485e2a46529dffae54c75ace (8ms) I0504 09:25:07.125375 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (24ms) I0504 09:25:07.226206 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (101ms) I0504 09:25:07.325778 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (100ms) I0504 09:25:07.426452 1 gather_workloads_info.go:387] No image sha256:9ef6dc311b880a6442b7a3a19006ddd955c7da36850bf000f5c9b68fd0862dd4 (101ms) I0504 09:25:07.525902 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (99ms) I0504 09:25:07.625199 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (99ms) I0504 09:25:07.727387 1 gather_workloads_info.go:387] No image sha256:5ff204630794311b0b37fa7b197b933ad85d76a481bf7cdb3bcbada08f0cdcbf (102ms) I0504 09:25:07.826356 1 gather_workloads_info.go:387] No image sha256:572ebf7c7fab1a518582f3ca46c866730423cce6e8a19f614dc15fd61bf0c65a (99ms) I0504 09:25:07.926290 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (100ms) I0504 09:25:08.026069 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (100ms) I0504 09:25:08.125847 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (100ms) I0504 09:25:08.225963 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (100ms) I0504 09:25:08.326090 1 gather_workloads_info.go:387] No image sha256:6c6d412ccf0c3d6a14645a79df6c4658c0f8725d7954cb83c3d2f0116499345a (100ms) I0504 09:25:08.426258 1 gather_workloads_info.go:387] No image sha256:15e1b5f763553e86d0230a751b0783e3b990438fd07cf96eed164bccb72ed564 (100ms) I0504 09:25:08.526763 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (100ms) I0504 09:25:08.627262 1 gather_workloads_info.go:387] No image sha256:d6fbe0075cbb12bfd287c973704fadf97154c7f73e370733d976a40835e9155a (100ms) I0504 09:25:08.726189 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (99ms) I0504 09:25:08.825851 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (100ms) I0504 09:25:08.926262 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (100ms) I0504 09:25:09.026170 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (100ms) I0504 09:25:09.126805 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (101ms) I0504 09:25:09.226460 1 gather_workloads_info.go:387] No image sha256:1eb6a6ee3c66ac15be6bc21bb6015c595c11c15b9dfd21bd9f5ce7f7b990ebaa (100ms) I0504 09:25:09.325787 1 gather_workloads_info.go:387] No image sha256:0b2e6536a0ca3afc5e64a218087a6287c74b77a359075a385f1949ca5bd51b22 (99ms) I0504 09:25:09.429981 1 gather_workloads_info.go:387] No image sha256:bdeb1c455cffccdb6c90db238e9d371de06340b1d6105d47f66d87293f878ac1 (104ms) I0504 09:25:09.525945 1 gather_workloads_info.go:387] No image sha256:9d6d63dce784df5863e6a0ba4c11317a26c9d756d4e62d47759c97190a71cce5 (96ms) I0504 09:25:09.625516 1 gather_workloads_info.go:387] No image sha256:15928a5b803fa00781b408dbc4c3bfa7c5287b18584ba31f817def8a4430c725 (100ms) I0504 09:25:09.725823 1 gather_workloads_info.go:387] No image sha256:17aefdc2f4752c23dc8f0947bb01fc34635822eeb18f2d8a5eddf8211b84d450 (100ms) I0504 09:25:09.829661 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (104ms) I0504 09:25:09.926054 1 gather_workloads_info.go:387] No image sha256:3380fecebaeee139c1607fbcd9346cebc92806679568e2244a8905ef0f52aad2 (96ms) I0504 09:25:10.026346 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (100ms) I0504 09:25:10.127075 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (101ms) I0504 09:25:10.225808 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (99ms) I0504 09:25:10.326569 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (101ms) I0504 09:25:10.425764 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (99ms) I0504 09:25:10.527280 1 gather_workloads_info.go:398] Found image sha256:23204877f166455ce8f9de9b91a467eeda576f172d5f96124a68aa986c4775d6 (101ms) I0504 09:25:10.625902 1 gather_workloads_info.go:387] No image sha256:48f5559439bea030a6423652da448c25fff0c62ba293eb521843c1a8eb89ddb0 (99ms) I0504 09:25:10.726870 1 gather_workloads_info.go:387] No image sha256:53ff8efdd8f563421ebc1859ab021a3bbc1306f82d4ff5d9c21bc60ec5f57898 (101ms) I0504 09:25:10.726904 1 tasks_processing.go:74] worker 1 stopped. I0504 09:25:10.727232 1 recorder.go:75] Recording config/workload_info with fingerprint=654c5d6ac23e96b20a57ffd48ee4aeb90e8db77ade5d50c29b1d61bd372476c2 I0504 09:25:10.727250 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.575405368s to process 1 records I0504 09:25:10.727265 1 periodic.go:263] Periodic gather workloads completed in 8.575s I0504 09:25:10.727273 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0504 09:25:10.727278 1 periodic.go:216] Running conditional gatherer I0504 09:25:10.732663 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0504 09:25:11.010308 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0504 09:25:12.057167 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0504 09:25:12.063284 1 conditional_gatherer.go:392] cluster version is '4.21.9' I0504 09:25:12.063409 1 tasks_processing.go:45] number of workers: 3 I0504 09:25:12.063452 1 tasks_processing.go:69] worker 2 listening for tasks. I0504 09:25:12.063485 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0504 09:25:12.063564 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 09:25:12.063563 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 09:25:12.063592 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0504 09:25:12.063614 1 tasks_processing.go:74] worker 1 stopped. I0504 09:25:12.063597 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0504 09:25:12.063648 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0504 09:25:12.063670 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.506µs to process 1 records I0504 09:25:12.063779 1 tasks_processing.go:74] worker 0 stopped. I0504 09:25:12.063799 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0504 09:25:12.063860 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 11.584µs to process 1 records I0504 09:25:12.064259 1 tasks_processing.go:74] worker 2 stopped. I0504 09:25:12.064277 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 621.362µs to process 0 records I0504 09:25:12.064312 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0504 09:25:12.064336 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0504 09:25:12.093494 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=e38119bfb4233e7b63b3ce56b750e2b2fefdb22ce1dfa4ae18d7889f901e9487 I0504 09:25:12.093632 1 diskrecorder.go:70] Writing 111 records to /var/lib/insights-operator/insights-2026-05-04-092512.tar.gz I0504 09:25:12.101238 1 diskrecorder.go:51] Wrote 111 records to disk in 7ms I0504 09:25:12.101271 1 periodic.go:285] Gathering cluster info every 2h0m0s I0504 09:25:12.101291 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0504 09:26:18.539692 1 diskrecorder.go:223] Found files to send: insights-2026-05-04-092512.tar.gz I0504 09:26:18.539728 1 insightsuploader.go:150] Checking archives to upload periodically every 16m15.692938258s I0504 09:26:18.539740 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0504 09:26:18.552886 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0504 09:26:18.734875 1 requests.go:87] Successfully reported id=2026-05-04T09:26:18Z x-rh-insights-request-id=f525d23eba23452ab88dd1e8059f9130, wrote=48808 I0504 09:26:18.734949 1 insightsuploader.go:187] Uploaded report successfully in 195.201012ms I0504 09:26:18.734971 1 controller.go:129] Initializing last reported time to 2026-05-04T09:26:18Z I0504 09:26:18.734992 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0504 09:26:18.735006 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0504 09:26:18.735016 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0504 09:26:18.740942 1 controller.go:554] The operator is healthy I0504 09:26:48.555014 1 controller.go:554] The operator is healthy I0504 09:27:19.690605 1 insightsreport.go:137] Pulling report from smart-proxy I0504 09:27:19.690628 1 insightsreport.go:149] Retrieving report I0504 09:27:19.697875 1 requests.go:110] Retrieving report for cluster: 62daaff4-6c7f-4f02-9812-f5356b9ed316 I0504 09:27:19.697890 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/62daaff4-6c7f-4f02-9812-f5356b9ed316/reports I0504 09:27:19.701108 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/62daaff4-6c7f-4f02-9812-f5356b9ed316/reports I0504 09:27:19.975686 1 insightsreport.go:184] Report retrieved I0504 09:27:20.001699 1 insightsreport.go:239] Report retrieved correctly I0504 09:28:48.552778 1 controller.go:554] The operator is healthy I0504 09:29:48.527401 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 09:29:48.533051 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 09:29:48.533070 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 09:29:48.537387 1 secretconfigobserver.go:119] support secret does not exist I0504 09:30:48.553241 1 controller.go:554] The operator is healthy I0504 09:32:48.553870 1 controller.go:554] The operator is healthy I0504 09:34:48.527322 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 09:34:48.531887 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 09:34:48.537958 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 09:34:48.542878 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 09:34:48.542898 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 09:34:48.547150 1 secretconfigobserver.go:119] support secret does not exist I0504 09:34:48.552145 1 controller.go:554] The operator is healthy I0504 09:36:48.556835 1 controller.go:554] The operator is healthy I0504 09:38:48.553083 1 controller.go:554] The operator is healthy I0504 09:39:48.547619 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 09:39:48.552627 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 09:39:48.552647 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 09:39:48.557076 1 secretconfigobserver.go:119] support secret does not exist I0504 09:40:48.553356 1 controller.go:554] The operator is healthy I0504 09:42:39.428601 1 insightsuploader.go:146] Nothing to report since 2026-05-04T09:26:18Z I0504 09:42:48.552061 1 controller.go:554] The operator is healthy I0504 09:44:48.525465 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 09:44:48.529603 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 09:44:48.551954 1 controller.go:554] The operator is healthy I0504 09:44:48.557818 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 09:44:48.562707 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 09:44:48.562721 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 09:44:48.566593 1 secretconfigobserver.go:119] support secret does not exist I0504 09:46:48.552014 1 controller.go:554] The operator is healthy I0504 09:48:48.553472 1 controller.go:554] The operator is healthy I0504 09:49:48.566925 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 09:49:48.572473 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 09:49:48.572490 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 09:49:48.576559 1 secretconfigobserver.go:119] support secret does not exist I0504 09:50:48.553403 1 controller.go:554] The operator is healthy I0504 09:52:48.553084 1 controller.go:554] The operator is healthy I0504 09:54:48.529490 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 09:54:48.552727 1 controller.go:554] The operator is healthy I0504 09:54:48.577394 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 09:54:48.582427 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 09:54:48.582451 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 09:54:48.587432 1 secretconfigobserver.go:119] support secret does not exist I0504 09:56:48.555839 1 controller.go:554] The operator is healthy I0504 09:58:38.367720 1 insightsuploader.go:146] Nothing to report since 2026-05-04T09:26:18Z