I0424 19:02:04.898537 1 cmd.go:253] Using service-serving-cert provided certificates I0424 19:02:04.898982 1 observer_polling.go:159] Starting file observer I0424 19:02:05.252687 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0424 19:02:05.252875 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0424 19:02:05.253464 1 secure_serving.go:57] Forcing use of http/1.1 only W0424 19:02:05.253490 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0424 19:02:05.253497 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0424 19:02:05.253504 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0424 19:02:05.253508 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0424 19:02:05.253512 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0424 19:02:05.253516 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0424 19:02:05.253518 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0424 19:02:05.257365 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0424 19:02:05.257418 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"8c937c8e-75a1-4118-a43f-fd8e4974e2ea", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0424 19:02:05.258206 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0424 19:02:05.258222 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0424 19:02:05.258228 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0424 19:02:05.258241 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0424 19:02:05.258222 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0424 19:02:05.258256 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0424 19:02:05.258496 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0424 19:02:05.258814 1 secure_serving.go:213] Serving securely on [::]:8443 I0424 19:02:05.258842 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0424 19:02:05.262457 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0424 19:02:05.262479 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0424 19:02:05.262569 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0424 19:02:05.266857 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 19:02:05.266873 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0424 19:02:05.270950 1 secretconfigobserver.go:119] support secret does not exist I0424 19:02:05.275248 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 19:02:05.278991 1 secretconfigobserver.go:119] support secret does not exist I0424 19:02:05.281129 1 recorder.go:176] Pruning old reports every 4h57m20s, max age is 288h0m0s I0424 19:02:05.285728 1 periodic.go:216] Running clusterconfig gatherer I0424 19:02:05.285744 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0424 19:02:05.285753 1 insightsreport.go:296] Starting report retriever I0424 19:02:05.285760 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0424 19:02:05.285777 1 tasks_processing.go:45] number of workers: 64 I0424 19:02:05.285801 1 tasks_processing.go:69] worker 1 listening for tasks. I0424 19:02:05.285812 1 tasks_processing.go:69] worker 2 listening for tasks. I0424 19:02:05.285816 1 tasks_processing.go:71] worker 1 working on qemu_kubevirt_launcher_logs task. I0424 19:02:05.285819 1 tasks_processing.go:69] worker 6 listening for tasks. I0424 19:02:05.285824 1 tasks_processing.go:69] worker 3 listening for tasks. I0424 19:02:05.285830 1 tasks_processing.go:69] worker 7 listening for tasks. I0424 19:02:05.285833 1 tasks_processing.go:69] worker 4 listening for tasks. I0424 19:02:05.285838 1 tasks_processing.go:69] worker 36 listening for tasks. I0424 19:02:05.285842 1 tasks_processing.go:69] worker 21 listening for tasks. I0424 19:02:05.285846 1 tasks_processing.go:69] worker 37 listening for tasks. I0424 19:02:05.285849 1 tasks_processing.go:69] worker 8 listening for tasks. I0424 19:02:05.285854 1 tasks_processing.go:69] worker 38 listening for tasks. I0424 19:02:05.285860 1 tasks_processing.go:69] worker 9 listening for tasks. I0424 19:02:05.285863 1 tasks_processing.go:69] worker 35 listening for tasks. I0424 19:02:05.285865 1 tasks_processing.go:69] worker 39 listening for tasks. I0424 19:02:05.285869 1 tasks_processing.go:69] worker 10 listening for tasks. I0424 19:02:05.285869 1 tasks_processing.go:69] worker 28 listening for tasks. I0424 19:02:05.285876 1 tasks_processing.go:69] worker 29 listening for tasks. I0424 19:02:05.285872 1 tasks_processing.go:69] worker 40 listening for tasks. I0424 19:02:05.285878 1 tasks_processing.go:69] worker 11 listening for tasks. I0424 19:02:05.285881 1 tasks_processing.go:69] worker 41 listening for tasks. I0424 19:02:05.285884 1 tasks_processing.go:69] worker 12 listening for tasks. I0424 19:02:05.285884 1 tasks_processing.go:69] worker 30 listening for tasks. I0424 19:02:05.285891 1 tasks_processing.go:69] worker 31 listening for tasks. I0424 19:02:05.285888 1 tasks_processing.go:69] worker 42 listening for tasks. I0424 19:02:05.285892 1 tasks_processing.go:69] worker 25 listening for tasks. I0424 19:02:05.285897 1 tasks_processing.go:69] worker 26 listening for tasks. I0424 19:02:05.285894 1 tasks_processing.go:69] worker 24 listening for tasks. I0424 19:02:05.285904 1 tasks_processing.go:69] worker 0 listening for tasks. I0424 19:02:05.285910 1 tasks_processing.go:69] worker 34 listening for tasks. I0424 19:02:05.285911 1 tasks_processing.go:69] worker 27 listening for tasks. I0424 19:02:05.285915 1 tasks_processing.go:69] worker 15 listening for tasks. I0424 19:02:05.285801 1 tasks_processing.go:69] worker 5 listening for tasks. I0424 19:02:05.285919 1 tasks_processing.go:69] worker 46 listening for tasks. I0424 19:02:05.285921 1 tasks_processing.go:69] worker 54 listening for tasks. I0424 19:02:05.285925 1 tasks_processing.go:69] worker 23 listening for tasks. I0424 19:02:05.285928 1 tasks_processing.go:69] worker 47 listening for tasks. I0424 19:02:05.285923 1 tasks_processing.go:71] worker 15 working on openshift_machine_api_events task. I0424 19:02:05.285935 1 tasks_processing.go:69] worker 17 listening for tasks. I0424 19:02:05.285937 1 tasks_processing.go:71] worker 6 working on openstack_controlplanes task. I0424 19:02:05.285940 1 tasks_processing.go:71] worker 23 working on olm_operators task. I0424 19:02:05.285943 1 tasks_processing.go:69] worker 60 listening for tasks. I0424 19:02:05.285940 1 tasks_processing.go:69] worker 33 listening for tasks. I0424 19:02:05.285951 1 tasks_processing.go:71] worker 60 working on nodes task. I0424 19:02:05.285911 1 tasks_processing.go:69] worker 45 listening for tasks. I0424 19:02:05.285729 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0424 19:02:05.285958 1 tasks_processing.go:69] worker 56 listening for tasks. I0424 19:02:05.285954 1 tasks_processing.go:69] worker 50 listening for tasks. I0424 19:02:05.285964 1 tasks_processing.go:69] worker 53 listening for tasks. I0424 19:02:05.285968 1 tasks_processing.go:69] worker 48 listening for tasks. I0424 19:02:05.285970 1 tasks_processing.go:69] worker 57 listening for tasks. I0424 19:02:05.285895 1 tasks_processing.go:69] worker 43 listening for tasks. I0424 19:02:05.285978 1 tasks_processing.go:71] worker 40 working on cost_management_metrics_configs task. I0424 19:02:05.285981 1 tasks_processing.go:71] worker 3 working on validating_webhook_configurations task. I0424 19:02:05.285985 1 tasks_processing.go:71] worker 12 working on cluster_apiserver task. I0424 19:02:05.285985 1 tasks_processing.go:71] worker 35 working on openstack_version task. I0424 19:02:05.285987 1 tasks_processing.go:71] worker 30 working on openshift_logging task. I0424 19:02:05.285983 1 tasks_processing.go:71] worker 29 working on service_accounts task. I0424 19:02:05.285996 1 tasks_processing.go:69] worker 61 listening for tasks. I0424 19:02:05.286007 1 tasks_processing.go:71] worker 21 working on lokistack task. I0424 19:02:05.286005 1 tasks_processing.go:71] worker 36 working on networks task. I0424 19:02:05.286181 1 tasks_processing.go:71] worker 25 working on pdbs task. I0424 19:02:05.286191 1 tasks_processing.go:71] worker 46 working on aggregated_monitoring_cr_names task. I0424 19:02:05.285814 1 tasks_processing.go:69] worker 63 listening for tasks. I0424 19:02:05.285935 1 tasks_processing.go:69] worker 59 listening for tasks. I0424 19:02:05.285943 1 tasks_processing.go:71] worker 17 working on dvo_metrics task. I0424 19:02:05.286225 1 tasks_processing.go:71] worker 37 working on certificate_signing_requests task. I0424 19:02:05.286236 1 tasks_processing.go:71] worker 7 working on support_secret task. I0424 19:02:05.286269 1 tasks_processing.go:71] worker 38 working on node_features task. I0424 19:02:05.286277 1 tasks_processing.go:71] worker 4 working on monitoring_persistent_volumes task. I0424 19:02:05.285930 1 tasks_processing.go:71] worker 2 working on machine_config_pools task. I0424 19:02:05.286336 1 tasks_processing.go:71] worker 26 working on ingress_certificates task. I0424 19:02:05.286418 1 tasks_processing.go:71] worker 39 working on container_images task. I0424 19:02:05.286460 1 tasks_processing.go:71] worker 33 working on schedulers task. I0424 19:02:05.285948 1 tasks_processing.go:71] worker 47 working on proxies task. I0424 19:02:05.286185 1 tasks_processing.go:71] worker 31 working on machine_autoscalers task. I0424 19:02:05.285898 1 tasks_processing.go:69] worker 13 listening for tasks. I0424 19:02:05.286462 1 tasks_processing.go:71] worker 43 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0424 19:02:05.285906 1 tasks_processing.go:69] worker 14 listening for tasks. I0424 19:02:05.286762 1 tasks_processing.go:71] worker 14 working on sap_config task. I0424 19:02:05.285957 1 tasks_processing.go:69] worker 52 listening for tasks. I0424 19:02:05.285947 1 tasks_processing.go:69] worker 55 listening for tasks. I0424 19:02:05.285957 1 tasks_processing.go:69] worker 20 listening for tasks. I0424 19:02:05.285962 1 tasks_processing.go:69] worker 19 listening for tasks. I0424 19:02:05.285965 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0424 19:02:05.285928 1 tasks_processing.go:71] worker 5 working on image task. I0424 19:02:05.285936 1 tasks_processing.go:69] worker 58 listening for tasks. I0424 19:02:05.285977 1 tasks_processing.go:71] worker 8 working on sap_pods task. I0424 19:02:05.285973 1 tasks_processing.go:69] worker 18 listening for tasks. I0424 19:02:05.287238 1 tasks_processing.go:71] worker 18 working on config_maps task. I0424 19:02:05.285976 1 tasks_processing.go:69] worker 49 listening for tasks. I0424 19:02:05.285978 1 tasks_processing.go:69] worker 32 listening for tasks. I0424 19:02:05.285986 1 tasks_processing.go:69] worker 62 listening for tasks. I0424 19:02:05.285925 1 tasks_processing.go:69] worker 16 listening for tasks. I0424 19:02:05.286432 1 tasks_processing.go:71] worker 54 working on install_plans task. I0424 19:02:05.287356 1 tasks_processing.go:71] worker 16 working on authentication task. I0424 19:02:05.287376 1 tasks_processing.go:71] worker 32 working on machine_sets task. I0424 19:02:05.287382 1 tasks_processing.go:71] worker 49 working on nodenetworkconfigurationpolicies task. I0424 19:02:05.287387 1 tasks_processing.go:71] worker 62 working on storage_cluster task. I0424 19:02:05.286440 1 tasks_processing.go:71] worker 11 working on container_runtime_configs task. I0424 19:02:05.286447 1 tasks_processing.go:71] worker 9 working on machine_configs task. I0424 19:02:05.285949 1 tasks_processing.go:69] worker 51 listening for tasks. I0424 19:02:05.287568 1 tasks_processing.go:71] worker 51 working on jaegers task. I0424 19:02:05.286451 1 tasks_processing.go:71] worker 59 working on operators_pods_and_events task. I0424 19:02:05.286455 1 tasks_processing.go:71] worker 48 working on machine_healthchecks task. I0424 19:02:05.286455 1 tasks_processing.go:71] worker 53 working on image_registries task. I0424 19:02:05.286458 1 tasks_processing.go:71] worker 57 working on metrics task. I0424 19:02:05.286467 1 tasks_processing.go:71] worker 61 working on oauths task. I0424 19:02:05.286470 1 tasks_processing.go:71] worker 63 working on version task. I0424 19:02:05.286476 1 tasks_processing.go:71] worker 56 working on infrastructures task. I0424 19:02:05.286480 1 tasks_processing.go:71] worker 45 working on silenced_alerts task. I0424 19:02:05.286486 1 tasks_processing.go:71] worker 41 working on sap_datahubs task. I0424 19:02:05.286492 1 tasks_processing.go:71] worker 50 working on openstack_dataplanenodesets task. I0424 19:02:05.286497 1 tasks_processing.go:71] worker 10 working on tsdb_status task. I0424 19:02:05.286503 1 tasks_processing.go:71] worker 28 working on storage_classes task. I0424 19:02:05.286509 1 tasks_processing.go:71] worker 34 working on feature_gates task. I0424 19:02:05.285904 1 tasks_processing.go:69] worker 44 listening for tasks. I0424 19:02:05.286512 1 tasks_processing.go:71] worker 24 working on mutating_webhook_configurations task. I0424 19:02:05.289474 1 tasks_processing.go:71] worker 44 working on image_pruners task. I0424 19:02:05.286518 1 tasks_processing.go:71] worker 27 working on overlapping_namespace_uids task. I0424 19:02:05.286525 1 tasks_processing.go:71] worker 0 working on clusterroles task. I0424 19:02:05.286697 1 tasks_processing.go:71] worker 13 working on nodenetworkstates task. I0424 19:02:05.285918 1 tasks_processing.go:69] worker 22 listening for tasks. I0424 19:02:05.286837 1 tasks_processing.go:71] worker 55 working on ingress task. I0424 19:02:05.289849 1 tasks_processing.go:71] worker 22 working on ceph_cluster task. I0424 19:02:05.286837 1 tasks_processing.go:71] worker 58 working on node_logs task. I0424 19:02:05.286841 1 tasks_processing.go:71] worker 52 working on active_alerts task. I0424 19:02:05.285974 1 tasks_processing.go:71] worker 42 working on pod_network_connectivity_checks task. I0424 19:02:05.286843 1 tasks_processing.go:71] worker 20 working on crds task. I0424 19:02:05.286847 1 tasks_processing.go:71] worker 19 working on openstack_dataplanedeployments task. I0424 19:02:05.289401 1 tasks_processing.go:71] worker 6 working on machines task. I0424 19:02:05.289407 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 3.451524ms to process 0 records I0424 19:02:05.290701 1 tasks_processing.go:71] worker 21 working on operators task. I0424 19:02:05.290704 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 3.922939ms to process 0 records I0424 19:02:05.291036 1 tasks_processing.go:74] worker 30 stopped. I0424 19:02:05.291053 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 5.030029ms to process 0 records I0424 19:02:05.291858 1 tasks_processing.go:74] worker 40 stopped. I0424 19:02:05.291870 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 5.869745ms to process 0 records E0424 19:02:05.310110 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0424 19:02:05.310127 1 tasks_processing.go:74] worker 38 stopped. I0424 19:02:05.310137 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 23.847544ms to process 0 records I0424 19:02:05.310151 1 tasks_processing.go:74] worker 31 stopped. I0424 19:02:05.310167 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 23.459814ms to process 0 records I0424 19:02:05.310264 1 tasks_processing.go:74] worker 47 stopped. I0424 19:02:05.310285 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0424 19:02:05.310303 1 controller.go:368] The initial operator extension status is healthy I0424 19:02:05.310312 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0424 19:02:05.310316 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0424 19:02:05.310319 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0424 19:02:05.310333 1 controller.go:531] The operator is still being initialized I0424 19:02:05.310339 1 controller.go:554] The operator is healthy I0424 19:02:05.310351 1 recorder.go:75] Recording config/proxy with fingerprint=afb9312cea3b189e4c5d56e8c19670fe761ff546c427da449ed6806dd6d9cef0 I0424 19:02:05.310366 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 23.652375ms to process 1 records I0424 19:02:05.310374 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 24.364695ms to process 0 records I0424 19:02:05.310382 1 tasks_processing.go:74] worker 35 stopped. I0424 19:02:05.310402 1 gather_logs.go:145] no pods in namespace were found I0424 19:02:05.310410 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 24.108962ms to process 0 records I0424 19:02:05.310404 1 tasks_processing.go:74] worker 2 stopped. I0424 19:02:05.310419 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 24.593548ms to process 0 records I0424 19:02:05.310423 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0424 19:02:05.310432 1 tasks_processing.go:74] worker 23 stopped. I0424 19:02:05.310435 1 tasks_processing.go:74] worker 1 stopped. I0424 19:02:05.310476 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0424 19:02:05.310427 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 24.470004ms to process 0 records I0424 19:02:05.310511 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 23.663967ms to process 0 records I0424 19:02:05.310519 1 tasks_processing.go:74] worker 14 stopped. W0424 19:02:05.310550 1 operator.go:328] started I0424 19:02:05.310572 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0424 19:02:05.310615 1 tasks_processing.go:74] worker 62 stopped. I0424 19:02:05.310627 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 23.220522ms to process 0 records I0424 19:02:05.310654 1 tasks_processing.go:74] worker 15 stopped. I0424 19:02:05.310672 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 24.710334ms to process 0 records I0424 19:02:05.310693 1 tasks_processing.go:74] worker 32 stopped. I0424 19:02:05.310701 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 23.304099ms to process 0 records I0424 19:02:05.310713 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 23.31648ms to process 0 records I0424 19:02:05.310717 1 tasks_processing.go:74] worker 49 stopped. I0424 19:02:05.311208 1 tasks_processing.go:74] worker 36 stopped. I0424 19:02:05.311399 1 recorder.go:75] Recording config/network with fingerprint=c5fdfaf83f7e2112d516dd589ad2b611020aaa441cb0385e54ecf89e40eecd09 I0424 19:02:05.311416 1 gather.go:177] gatherer "clusterconfig" function "networks" took 25.187941ms to process 1 records I0424 19:02:05.311792 1 tasks_processing.go:74] worker 4 stopped. I0424 19:02:05.311807 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 25.504468ms to process 0 records I0424 19:02:05.311819 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 21.849104ms to process 0 records E0424 19:02:05.311828 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0424 19:02:05.311831 1 tasks_processing.go:74] worker 22 stopped. I0424 19:02:05.311840 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 21.722048ms to process 0 records I0424 19:02:05.311857 1 tasks_processing.go:74] worker 42 stopped. I0424 19:02:05.311967 1 tasks_processing.go:74] worker 12 stopped. I0424 19:02:05.312016 1 recorder.go:75] Recording config/apiserver with fingerprint=fc3ef5757221eebda81202ffcfb54a39bcf9dc912560f2741feb524380c284e8 I0424 19:02:05.312031 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 25.826117ms to process 1 records I0424 19:02:05.312617 1 tasks_processing.go:74] worker 50 stopped. I0424 19:02:05.312629 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 24.30129ms to process 0 records W0424 19:02:05.312898 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0424 19:02:05.315331 1 tasks_processing.go:74] worker 11 stopped. I0424 19:02:05.315343 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 27.859865ms to process 0 records I0424 19:02:05.315719 1 tasks_processing.go:74] worker 51 stopped. I0424 19:02:05.315732 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 28.14259ms to process 0 records I0424 19:02:05.315791 1 tasks_processing.go:74] worker 28 stopped. I0424 19:02:05.315885 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=8498298a9d7688fb9d75a36e6c462dc0016e15e19e7fd80ec3c684e445a5b72f I0424 19:02:05.315902 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=9c54f9974df1c03a265753d15b71b58d5b38a52e10d0de0979bd6251943ec0e7 I0424 19:02:05.315907 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 27.166941ms to process 2 records I0424 19:02:05.315943 1 tasks_processing.go:74] worker 7 stopped. E0424 19:02:05.315953 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0424 19:02:05.315964 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 29.693202ms to process 0 records I0424 19:02:05.316025 1 tasks_processing.go:74] worker 5 stopped. I0424 19:02:05.316284 1 recorder.go:75] Recording config/image with fingerprint=24fb2f43a045c54b40d3d5623cd0153b7fc0d83bea657dd5498beef5c53463b1 I0424 19:02:05.316353 1 gather.go:177] gatherer "clusterconfig" function "image" took 29.214912ms to process 1 records I0424 19:02:05.317093 1 tasks_processing.go:74] worker 16 stopped. I0424 19:02:05.318202 1 recorder.go:75] Recording config/authentication with fingerprint=d6f1ec8ab168676a42ab8a5af60fa3fe11c8175fdd224fda0cf27d13c5191aaf I0424 19:02:05.318423 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 28.701841ms to process 1 records I0424 19:02:05.318565 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=d9a9d7cc7ae7d3043a5dcdcb2c0574a193831b15a579db04c293b56c71812c87 I0424 19:02:05.318601 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 29.598253ms to process 1 records I0424 19:02:05.318629 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 26.42426ms to process 0 records I0424 19:02:05.318891 1 tasks_processing.go:74] worker 13 stopped. I0424 19:02:05.319035 1 tasks_processing.go:74] worker 3 stopped. I0424 19:02:05.318972 1 tasks_processing.go:74] worker 33 stopped. I0424 19:02:05.319464 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=4f5bdbd7daf8606bd0e0bb0c32c0368b05bf484949ac7dc057b24a42898a1f02 I0424 19:02:05.319633 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=df8fd5753604d228a4284f842b8ac26c6b22d67b76172802dd87349a918daf56 I0424 19:02:05.320370 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=72ddb04de69f111e557f59f43ce9c7b50c0f14de0ac3545932f4719d4d30216d I0424 19:02:05.320435 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=f6015b8b8757bff62f4e4f4db37a3ca8edfa149db9f3c6e04d52bfe9d51d1142 I0424 19:02:05.320479 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=01ee22959879ee5cc9a0a69817636ba29f50af77004d49d3af89daa37f889426 I0424 19:02:05.320524 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=7a99813f198fafd996fdef79ee11a4d93b6758c492ec45e7f9a6c69aa3335fce I0424 19:02:05.320571 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=be1e594ec697c41ade9d130af4d78e43e7b99507761502e5d7850c8abb2113f0 I0424 19:02:05.320624 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=a422ac03803beddb06e9b73a1aa4d5a1449dbd80c304f69e177805dedfdb1db3 I0424 19:02:05.320673 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=426cac7951186226d280ce96fbce97272c193f029d6a04557315e25c70d6c1bd I0424 19:02:05.320777 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=5601431b9c07334083df6a0c93018a75220096f8e13eca25deb4afa24fdb59e2 I0424 19:02:05.320820 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=a21ec3b80823c925c664212c8bf75bab0eae6af08cfeac71b57415dd06d628d5 I0424 19:02:05.320863 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=a54a0b836f8924ffa18b23f4ddfce28cc668884a1f36c9273386a6a1b465f4ae I0424 19:02:05.320932 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=82b117488322850e5dea8f09d78a792b7addf34002fc02bd8b99f76c36ac1ada I0424 19:02:05.320946 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 30.367423ms to process 13 records I0424 19:02:05.321000 1 controller.go:254] Source scaController *sca.Controller is not ready I0424 19:02:05.321014 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0424 19:02:05.321020 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0424 19:02:05.321026 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0424 19:02:05.321030 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0424 19:02:05.321041 1 tasks_processing.go:74] worker 60 stopped. I0424 19:02:05.321056 1 controller.go:531] The operator is still being initialized I0424 19:02:05.321065 1 controller.go:554] The operator is healthy I0424 19:02:05.321473 1 recorder.go:75] Recording config/node/ip-10-0-0-135.ec2.internal with fingerprint=118f5bb35fd98da09a717ea29d61640723ee7a32dd30ff0736bc6fceefdbff5e I0424 19:02:05.321582 1 recorder.go:75] Recording config/node/ip-10-0-1-203.ec2.internal with fingerprint=daceb738baca929c543d931055fd46cd8e383adc815b3af9103d17fc0f2d69df I0424 19:02:05.321689 1 recorder.go:75] Recording config/node/ip-10-0-2-94.ec2.internal with fingerprint=05cbb620f9a0efa805c5af1c36eea35fba09b3706f44901a9299f899e64f697a I0424 19:02:05.321704 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 30.413961ms to process 3 records I0424 19:02:05.321716 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 26.740439ms to process 0 records I0424 19:02:05.321736 1 tasks_processing.go:74] worker 19 stopped. I0424 19:02:05.321811 1 tasks_processing.go:74] worker 44 stopped. I0424 19:02:05.322100 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=aa01154773528d3c67746c75d9e8e90bd92726c9d524c2906db4f7636a35a84f I0424 19:02:05.322119 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 27.826518ms to process 1 records E0424 19:02:05.322131 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0424 19:02:05.322144 1 gather.go:177] gatherer "clusterconfig" function "machines" took 27.189605ms to process 0 records I0424 19:02:05.322156 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 30.355514ms to process 0 records I0424 19:02:05.322162 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 29.432661ms to process 0 records I0424 19:02:05.322164 1 tasks_processing.go:74] worker 6 stopped. I0424 19:02:05.322175 1 tasks_processing.go:74] worker 41 stopped. I0424 19:02:05.322173 1 tasks_processing.go:74] worker 8 stopped. I0424 19:02:05.322244 1 tasks_processing.go:74] worker 55 stopped. I0424 19:02:05.322347 1 recorder.go:75] Recording config/ingress with fingerprint=b11bf592035989c1c0edf26033f59c20b05fde9e77fc59027e5783b6c5683d08 I0424 19:02:05.322367 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 27.822889ms to process 1 records I0424 19:02:05.322450 1 tasks_processing.go:74] worker 25 stopped. I0424 19:02:05.322482 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=098203609fed4f2d0f3d1c2fc3d721312aa180af50dc984e8a92bbf3dfb104da I0424 19:02:05.322514 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=b68d665853b5db24db8c53caf73200fe9fba29aeb075732867118fe1378331ab I0424 19:02:05.322577 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=bc7fd50c14468d8a03d61a1beec7f848c62452659bb8d94997b834cc36ade5ee I0424 19:02:05.322619 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=406dcc4b90403216e489f1227ae6131f25b8826c012a20da0b789000f0fc6d2a I0424 19:02:05.322650 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=6b520d2cad3467861300b334f8edb2be5221602e930061a0890aff0fadb23000 I0424 19:02:05.322734 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=83565db88cd7733993ed3b60670a5adbb21a9462699b4a9468d305271ac21cc8 I0424 19:02:05.322769 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=a4bc44e0ea5ad1d60aa4b75fd5429ccee15b7592213f0a5da0cfd60a7e062d4b I0424 19:02:05.322799 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=ffabca2940bcf8e17df1368cd8913923521d342a9f032a8f58d0f60f18ef8a60 I0424 19:02:05.322838 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=816ec97dc3579683a6becb71d6fd41b8e99b6ee776ac5afb5a118b619773718a I0424 19:02:05.322872 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=887e8b2b84a21428c374aa53fe75fd2d1fe83d648fdddaa53b8f0a67aff31cbc I0424 19:02:05.322902 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=9b97f6d680434afe49b042e4ebea9cc0d79ce591e84564d0000bb802c742a41a I0424 19:02:05.322918 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 31.617487ms to process 11 records E0424 19:02:05.322930 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0424 19:02:05.322942 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 30.369136ms to process 0 records I0424 19:02:05.322967 1 tasks_processing.go:74] worker 48 stopped. I0424 19:02:05.323044 1 tasks_processing.go:74] worker 34 stopped. I0424 19:02:05.323250 1 recorder.go:75] Recording config/featuregate with fingerprint=c4cda2a5da1fe604d30155d4c89a2553f6bbacab9e2fe253a762e43bbd954ad5 I0424 19:02:05.323270 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 29.547212ms to process 1 records I0424 19:02:05.323367 1 tasks_processing.go:74] worker 53 stopped. I0424 19:02:05.324037 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=4dc62b3fd3499e4cd1e88a31424c28ed3bb535d07b745e82a14c027772bdf419 I0424 19:02:05.324058 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 30.86008ms to process 1 records I0424 19:02:05.324178 1 tasks_processing.go:74] worker 61 stopped. I0424 19:02:05.324423 1 recorder.go:75] Recording config/oauth with fingerprint=8eb5edbea5aaf8efd7ae4c178059c83edd3f792a4a20c3f021431419f65134fc I0424 19:02:05.324486 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 31.420067ms to process 1 records I0424 19:02:05.324636 1 tasks_processing.go:74] worker 56 stopped. I0424 19:02:05.326188 1 recorder.go:75] Recording config/infrastructure with fingerprint=08ecb79ef56d82e6e5006649e686f7d31488e42ea1d05dc2e11970e000ec377e I0424 19:02:05.326260 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 31.532022ms to process 1 records E0424 19:02:05.326295 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0424 19:02:05.326339 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0424 19:02:05.326374 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 33.369262ms to process 1 records I0424 19:02:05.326410 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 35.135572ms to process 0 records I0424 19:02:05.326436 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 32.056038ms to process 0 records I0424 19:02:05.326584 1 tasks_processing.go:74] worker 37 stopped. I0424 19:02:05.326596 1 tasks_processing.go:74] worker 58 stopped. I0424 19:02:05.326676 1 tasks_processing.go:74] worker 24 stopped. I0424 19:02:05.326602 1 tasks_processing.go:74] worker 9 stopped. I0424 19:02:05.326814 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=8360f8a61f5f4d7bb6bfb9944ab85a3f2f158646154170b31cf1431d5cc2b663 I0424 19:02:05.326918 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=2dd338ec7c850645c2a7398f8f54d2088be1c69702001cacaea47b61526ab57e I0424 19:02:05.327011 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=951d91920d16c4734cee4eaa584edf7a46f4bcea1505a7059da00297e65e6eeb I0424 19:02:05.327029 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 33.420617ms to process 3 records I0424 19:02:05.327051 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 39.134742ms to process 0 records I0424 19:02:05.327095 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0424 19:02:05.327104 1 tasks_processing.go:74] worker 46 stopped. I0424 19:02:05.327118 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 36.640785ms to process 1 records I0424 19:02:05.327115 1 tasks_processing.go:74] worker 27 stopped. I0424 19:02:05.327413 1 tasks_processing.go:74] worker 63 stopped. I0424 19:02:05.328098 1 recorder.go:75] Recording config/version with fingerprint=d24d21c1cba8674b256ff40cf06b8e202bd63cb4abc296945f77b0fb1b97ffc0 I0424 19:02:05.328183 1 recorder.go:75] Recording config/id with fingerprint=2f8b7e0eb83d6b9eec4e3b01fc33b77c27c79fa2da4b1367dcb974925b17d44a I0424 19:02:05.328197 1 gather.go:177] gatherer "clusterconfig" function "version" took 39.538832ms to process 2 records I0424 19:02:05.329975 1 tasks_processing.go:74] worker 59 stopped. I0424 19:02:05.329990 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 42.383733ms to process 0 records I0424 19:02:05.330710 1 tasks_processing.go:74] worker 0 stopped. I0424 19:02:05.330953 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=e0849019b7ed9e6a84987669fc72bc707b19b3ef438cdb34ee374fc4262b51cd I0424 19:02:05.331133 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=eb9a40150874afa9e9143424614b25ed07c994d9afed019adfeb531bc9c1b0e0 I0424 19:02:05.331150 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 41.121302ms to process 2 records I0424 19:02:05.331492 1 tasks_processing.go:74] worker 20 stopped. I0424 19:02:05.332197 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=e68b3f34178384c3b2a7998eeecdb7b82d1e1b095b5033f6d1e125917092d575 I0424 19:02:05.332544 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=54fc7933d4de851909b8e7fc446f9d1ed56be0b1eda8d45ad6b05e92f07b3dec I0424 19:02:05.332562 1 gather.go:177] gatherer "clusterconfig" function "crds" took 41.30295ms to process 2 records I0424 19:02:05.336172 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0424 19:02:05.345725 1 tasks_processing.go:74] worker 39 stopped. I0424 19:02:05.347330 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-66dbbb6468-4zf6t with fingerprint=0ffab1d7c72415c39ba3d50017064893ea7b97bb3fb2fc2ccb73a0456b7229f7 I0424 19:02:05.347429 1 recorder.go:75] Recording config/running_containers with fingerprint=d1494d2e8cc2894918afe569b94046bdaf4a13212dc7fbb7b19246770fbde62a I0424 19:02:05.347441 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 59.25141ms to process 2 records I0424 19:02:05.347986 1 tasks_processing.go:74] worker 43 stopped. I0424 19:02:05.348003 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 61.274079ms to process 0 records I0424 19:02:05.350824 1 tasks_processing.go:74] worker 18 stopped. E0424 19:02:05.350839 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0424 19:02:05.350850 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0424 19:02:05.350854 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0424 19:02:05.350865 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0424 19:02:05.350904 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0424 19:02:05.350915 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0424 19:02:05.350938 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0424 19:02:05.350953 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0424 19:02:05.350991 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0424 19:02:05.350998 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0424 19:02:05.351003 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 63.571058ms to process 7 records I0424 19:02:05.358385 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0424 19:02:05.358397 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0424 19:02:05.358415 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0424 19:02:05.362690 1 base_controller.go:82] Caches are synced for ConfigController I0424 19:02:05.362703 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0424 19:02:05.366918 1 configmapobserver.go:84] configmaps "insights-config" not found I0424 19:02:05.411127 1 base_controller.go:82] Caches are synced for LoggingSyncer I0424 19:02:05.411143 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0424 19:02:05.419780 1 cluster_transfer.go:102] no available accepted cluster transfer I0424 19:02:05.419793 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0424 19:02:05.432456 1 tasks_processing.go:74] worker 26 stopped. E0424 19:02:05.432474 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0424 19:02:05.432487 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2psmoi83a7tsfb6q4lqvt94cau3j4e95-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2psmoi83a7tsfb6q4lqvt94cau3j4e95-primary-cert-bundle-secret" not found I0424 19:02:05.432561 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=fd517bbe565e62e3dbeb38b5d419afa3d67ccb0de653e1266a72ed432ad9ed2e I0424 19:02:05.432579 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 146.06543ms to process 1 records I0424 19:02:05.533720 1 sca.go:266] etc-pki-entitlement secret successfully created I0424 19:02:05.533738 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated W0424 19:02:06.312659 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0424 19:02:06.363205 1 tasks_processing.go:74] worker 52 stopped. I0424 19:02:06.363252 1 recorder.go:75] Recording config/alerts with fingerprint=0d64fc7328eead78c498fc577bd5a74c021322135a6ab58eecad193805a68d16 I0424 19:02:06.363271 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.073150236s to process 1 records I0424 19:02:06.363302 1 tasks_processing.go:74] worker 45 stopped. I0424 19:02:06.363317 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0424 19:02:06.363328 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.0752707s to process 1 records I0424 19:02:06.378776 1 tasks_processing.go:74] worker 57 stopped. I0424 19:02:06.378808 1 recorder.go:75] Recording config/metrics with fingerprint=c0c0f9be3c6131454c55396214604dbf5f750431160a187949ecbdcafffff746 I0424 19:02:06.378816 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 1.090987101s to process 1 records I0424 19:02:06.383668 1 tasks_processing.go:74] worker 10 stopped. I0424 19:02:06.383686 1 recorder.go:75] Recording config/tsdb.json with fingerprint=fc13a43cef40e28dcc100a14515e58e7e7a8953cbb40b05705da8d158f071de7 I0424 19:02:06.383695 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 1.095217418s to process 1 records W0424 19:02:07.312938 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0424 19:02:07.747699 1 tasks_processing.go:74] worker 21 stopped. I0424 19:02:07.747813 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=4968969cbe411ddb84cefb631b6e2a703b0cbbc917435ae88f9bf2e094a55731 I0424 19:02:07.747861 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0424 19:02:07.747894 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=9ad22f89e48537b04b40405e036889319fff4fe9b302cc7dbe1f535f203b4743 I0424 19:02:07.747905 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0424 19:02:07.747925 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=c7438b080ff895e9f817668317544edf66df6b9475efb1c08acb23d9d42e738a I0424 19:02:07.747941 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0424 19:02:07.747961 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=677f6e5b1d97aa7f4c052476766312a634f2c63b3cfd19f174a18fd94043d535 I0424 19:02:07.747988 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=da95a8443f63d07f72c75f5f01be1f8672e3b6cb20f2cfc4445c40df83a2584b I0424 19:02:07.748009 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=c920d159a41b32cf1451e14456d97045fc9c12902e649145bae0458145834c1b I0424 19:02:07.748038 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=fb919cccad606d478a25557864d21250dc62ed6db0e7cea1a78c047b3fed74af I0424 19:02:07.748345 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=7f52c89bbe4b4ed5e485fdd9461788e48fce1f91c2c01f5fa90220a932a69c21 I0424 19:02:07.748369 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=1ef5ad5cb8dc944b40d28ebd4c986556e86e108998ed7df3424d45cc88bac388 I0424 19:02:07.748379 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0424 19:02:07.748394 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=f47e28a6e37e38cee65a11309f9443be0b438a8e3563286cbc91079b8f3a87ff I0424 19:02:07.748405 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0424 19:02:07.748421 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=064737000ad72527d54fbf9b98e4b93394c65b883b8f609fcb9c755f4f202ec4 I0424 19:02:07.748430 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0424 19:02:07.748449 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=883a40c684c36651eb929db275c458e5eabf8c7f2d5ceae3f62d55f18487a9b7 I0424 19:02:07.748460 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0424 19:02:07.748482 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=0bbdd03faf9fead6fc49b9d05f14c35bb8ea8ad1ae249d2254832856a22733a4 I0424 19:02:07.748586 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=531e5e5154b82d726d3ab67ce49d93d11346c11f782861d8e8296a36d04053b7 I0424 19:02:07.748595 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0424 19:02:07.748605 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0424 19:02:07.748625 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0424 19:02:07.748648 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=af7b00246bf07098fe60b00f9a77702d9e4d3779fe2d8901a1d7d84d8cb75f3a I0424 19:02:07.748672 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=a4425a98012d7cca3059cd871c6122d5eea607b665b5de89af786e0f8d786117 I0424 19:02:07.748681 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0424 19:02:07.748697 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=e001c90e6b36bcdaa88f2dea04861fcedb972c597891efd920c9d978c0c22fb2 I0424 19:02:07.748713 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0424 19:02:07.748730 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=95f9d157020e497bc38ec0b442d0062495148d65bf23b25a37134833ab074e63 I0424 19:02:07.748740 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0424 19:02:07.748752 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=dea66703ef7a492a25883002a39fcdd14b645d34a12d68804d7a51424efe1d6f I0424 19:02:07.748767 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=e15c9deb6b91c04b5d7f51557ddc08edfc38c99077044bef07dfc3004624f12d I0424 19:02:07.748780 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=e12a9f7fa338757bb2add191f6fa59e733506b15df4a843f0ef54901fb0b87d9 I0424 19:02:07.748802 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=5d4bb20b52f57d9521a654c34f5b57ee002e5265793acf2aecba7754726f2035 I0424 19:02:07.748811 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0424 19:02:07.748837 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=db0dd48d1fcd1c732e41660ebec70bd28138eacd4a1e9e989ef72d067700cc4f I0424 19:02:07.748855 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0424 19:02:07.748861 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0424 19:02:07.748868 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.456976723s to process 39 records W0424 19:02:08.312818 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0424 19:02:09.312837 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0424 19:02:10.310979 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0424 19:02:10.311006 1 tasks_processing.go:74] worker 17 stopped. E0424 19:02:10.311024 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0424 19:02:10.311038 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0424 19:02:10.311050 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.024778167s to process 1 records I0424 19:02:18.137929 1 tasks_processing.go:74] worker 54 stopped. I0424 19:02:18.137969 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0424 19:02:18.137982 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.850608056s to process 1 records I0424 19:02:18.891330 1 tasks_processing.go:74] worker 29 stopped. I0424 19:02:18.891614 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=80e5b054934e37ddf232a6b52185c88ef3149965af63f70b0125fd661e8f707e I0424 19:02:18.891634 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.605317589s to process 1 records E0424 19:02:18.891697 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.605s with: function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"machines\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0424 19:02:18.892804 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "machines" failed with an error, function "machine_healthchecks" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0424 19:02:18.892820 1 periodic.go:216] Running workloads gatherer I0424 19:02:18.892839 1 tasks_processing.go:45] number of workers: 2 I0424 19:02:18.892847 1 tasks_processing.go:69] worker 1 listening for tasks. I0424 19:02:18.892851 1 tasks_processing.go:71] worker 1 working on workload_info task. I0424 19:02:18.892859 1 tasks_processing.go:69] worker 0 listening for tasks. I0424 19:02:18.892877 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0424 19:02:18.898796 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-94.ec2.internal... I0424 19:02:18.898814 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-203.ec2.internal... I0424 19:02:18.898799 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-135.ec2.internal... I0424 19:02:18.914691 1 tasks_processing.go:74] worker 0 stopped. I0424 19:02:18.914707 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 21.798827ms to process 0 records I0424 19:02:23.265013 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 4s I0424 19:02:23.287894 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0424 19:02:23.314732 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (28ms) I0424 19:02:23.322945 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (8ms) I0424 19:02:23.330916 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (8ms) I0424 19:02:23.338530 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (8ms) I0424 19:02:23.345973 1 gather_workloads_info.go:387] No image sha256:17aefdc2f4752c23dc8f0947bb01fc34635822eeb18f2d8a5eddf8211b84d450 (7ms) I0424 19:02:23.353134 1 gather_workloads_info.go:387] No image sha256:5ff204630794311b0b37fa7b197b933ad85d76a481bf7cdb3bcbada08f0cdcbf (7ms) I0424 19:02:23.360485 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (7ms) I0424 19:02:23.367696 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (7ms) I0424 19:02:23.375490 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (8ms) I0424 19:02:23.383323 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (8ms) I0424 19:02:23.395604 1 gather_workloads_info.go:387] No image sha256:0acfe677ed58031f8381cb8a4f2d8917ec4cfa95485e2a46529dffae54c75ace (12ms) I0424 19:02:23.496402 1 gather_workloads_info.go:387] No image sha256:572ebf7c7fab1a518582f3ca46c866730423cce6e8a19f614dc15fd61bf0c65a (101ms) I0424 19:02:23.597201 1 gather_workloads_info.go:387] No image sha256:15e1b5f763553e86d0230a751b0783e3b990438fd07cf96eed164bccb72ed564 (101ms) I0424 19:02:23.696455 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (99ms) I0424 19:02:23.797022 1 gather_workloads_info.go:398] Found image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (101ms) I0424 19:02:23.896771 1 gather_workloads_info.go:387] No image sha256:6c6d412ccf0c3d6a14645a79df6c4658c0f8725d7954cb83c3d2f0116499345a (100ms) I0424 19:02:23.996472 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (100ms) I0424 19:02:24.097271 1 gather_workloads_info.go:387] No image sha256:9d6d63dce784df5863e6a0ba4c11317a26c9d756d4e62d47759c97190a71cce5 (101ms) I0424 19:02:24.196687 1 gather_workloads_info.go:387] No image sha256:52db05084f2a2e7d7bf4040e4865df88ec4074efbbe59a78d861be2c5ab2aa40 (99ms) I0424 19:02:24.297117 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (100ms) I0424 19:02:24.396701 1 gather_workloads_info.go:398] Found image sha256:23204877f166455ce8f9de9b91a467eeda576f172d5f96124a68aa986c4775d6 (100ms) I0424 19:02:24.496240 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (99ms) I0424 19:02:24.596736 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (100ms) I0424 19:02:24.696340 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (100ms) I0424 19:02:24.796519 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (100ms) I0424 19:02:24.896869 1 gather_workloads_info.go:387] No image sha256:0b2e6536a0ca3afc5e64a218087a6287c74b77a359075a385f1949ca5bd51b22 (100ms) I0424 19:02:24.997293 1 gather_workloads_info.go:387] No image sha256:9ef6dc311b880a6442b7a3a19006ddd955c7da36850bf000f5c9b68fd0862dd4 (100ms) I0424 19:02:25.101596 1 gather_workloads_info.go:387] No image sha256:15928a5b803fa00781b408dbc4c3bfa7c5287b18584ba31f817def8a4430c725 (104ms) I0424 19:02:25.197013 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (95ms) I0424 19:02:25.297203 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (100ms) I0424 19:02:25.396759 1 gather_workloads_info.go:387] No image sha256:5940cc34b322325618929b1e0eb3864dae73d54c8f5b474e218dfb28123a61c7 (100ms) I0424 19:02:25.496373 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (100ms) I0424 19:02:25.596978 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (101ms) I0424 19:02:25.696119 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (99ms) I0424 19:02:25.796790 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (101ms) I0424 19:02:25.896272 1 gather_workloads_info.go:387] No image sha256:5d6a2b8d2508c7da71192661e7c0dcf1335f48b6336a20189abffbe79eeff962 (99ms) I0424 19:02:25.997615 1 gather_workloads_info.go:387] No image sha256:53ff8efdd8f563421ebc1859ab021a3bbc1306f82d4ff5d9c21bc60ec5f57898 (101ms) I0424 19:02:26.096590 1 gather_workloads_info.go:387] No image sha256:48f5559439bea030a6423652da448c25fff0c62ba293eb521843c1a8eb89ddb0 (99ms) I0424 19:02:26.195791 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (99ms) I0424 19:02:26.296928 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (101ms) I0424 19:02:26.396671 1 gather_workloads_info.go:387] No image sha256:1eb6a6ee3c66ac15be6bc21bb6015c595c11c15b9dfd21bd9f5ce7f7b990ebaa (100ms) I0424 19:02:26.496270 1 gather_workloads_info.go:387] No image sha256:3380fecebaeee139c1607fbcd9346cebc92806679568e2244a8905ef0f52aad2 (100ms) I0424 19:02:26.596518 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (100ms) I0424 19:02:26.697484 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (101ms) I0424 19:02:26.796654 1 gather_workloads_info.go:387] No image sha256:bdeb1c455cffccdb6c90db238e9d371de06340b1d6105d47f66d87293f878ac1 (99ms) I0424 19:02:26.896765 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (100ms) I0424 19:02:26.896797 1 tasks_processing.go:74] worker 1 stopped. I0424 19:02:26.897143 1 recorder.go:75] Recording config/workload_info with fingerprint=0d1eccc626dabaf8f9c8b82fb5c727876b0b64cfaffdc2cf402b101c9fae3396 I0424 19:02:26.897162 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.003938654s to process 1 records I0424 19:02:26.897174 1 periodic.go:263] Periodic gather workloads completed in 8.004s I0424 19:02:26.897182 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0424 19:02:26.897187 1 periodic.go:216] Running conditional gatherer I0424 19:02:26.902958 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0424 19:02:27.179744 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0424 19:02:27.182018 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0424 19:02:27.188024 1 conditional_gatherer.go:392] cluster version is '4.21.9' I0424 19:02:27.188052 1 tasks_processing.go:45] number of workers: 3 I0424 19:02:27.188059 1 tasks_processing.go:69] worker 2 listening for tasks. I0424 19:02:27.188064 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0424 19:02:27.188091 1 tasks_processing.go:69] worker 1 listening for tasks. I0424 19:02:27.188086 1 tasks_processing.go:69] worker 0 listening for tasks. I0424 19:02:27.188103 1 tasks_processing.go:71] worker 1 working on remote_configuration task. I0424 19:02:27.188108 1 tasks_processing.go:74] worker 0 stopped. I0424 19:02:27.188109 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0424 19:02:27.188158 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0424 19:02:27.188176 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 914ns to process 1 records I0424 19:02:27.188214 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0424 19:02:27.188222 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.199µs to process 1 records I0424 19:02:27.188228 1 tasks_processing.go:74] worker 1 stopped. I0424 19:02:27.188370 1 tasks_processing.go:74] worker 2 stopped. I0424 19:02:27.188386 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 254.584µs to process 0 records I0424 19:02:27.188417 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0424 19:02:27.188447 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0424 19:02:27.213038 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=24e3c851e91f021124f7179d7cdb1831e0defda74f4528eef53ce04cb257fe17 I0424 19:02:27.213248 1 diskrecorder.go:70] Writing 112 records to /var/lib/insights-operator/insights-2026-04-24-190227.tar.gz I0424 19:02:27.220969 1 diskrecorder.go:51] Wrote 112 records to disk in 7ms I0424 19:02:27.221004 1 periodic.go:285] Gathering cluster info every 2h0m0s I0424 19:02:27.221025 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0424 19:03:35.287791 1 diskrecorder.go:223] Found files to send: insights-2026-04-24-190227.tar.gz I0424 19:03:35.287820 1 insightsuploader.go:150] Checking archives to upload periodically every 15m44.856465874s I0424 19:03:35.287828 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0424 19:03:35.299334 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0424 19:03:35.489617 1 requests.go:87] Successfully reported id=2026-04-24T19:03:35Z x-rh-insights-request-id=5e8a386c34ed46a7bc10c2e495080862, wrote=49481 I0424 19:03:35.489677 1 insightsuploader.go:187] Uploaded report successfully in 201.841704ms I0424 19:03:35.489698 1 controller.go:129] Initializing last reported time to 2026-04-24T19:03:35Z I0424 19:03:35.489774 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0424 19:03:35.489785 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0424 19:03:35.489793 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0424 19:03:35.494911 1 controller.go:554] The operator is healthy I0424 19:04:05.316757 1 controller.go:554] The operator is healthy I0424 19:04:41.282383 1 insightsreport.go:137] Pulling report from smart-proxy I0424 19:04:41.282409 1 insightsreport.go:149] Retrieving report I0424 19:04:41.289227 1 requests.go:110] Retrieving report for cluster: 8f8e6174-9178-4560-b705-69b1c13d18ac I0424 19:04:41.289240 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/8f8e6174-9178-4560-b705-69b1c13d18ac/reports I0424 19:04:41.292305 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/8f8e6174-9178-4560-b705-69b1c13d18ac/reports I0424 19:04:41.474020 1 insightsreport.go:184] Report retrieved I0424 19:04:41.500780 1 insightsreport.go:239] Report retrieved correctly I0424 19:06:05.319273 1 controller.go:554] The operator is healthy I0424 19:07:05.279385 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0424 19:07:05.293410 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 19:07:05.293428 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0424 19:07:05.300162 1 secretconfigobserver.go:119] support secret does not exist I0424 19:08:05.319458 1 controller.go:554] The operator is healthy I0424 19:10:05.317587 1 controller.go:554] The operator is healthy I0424 19:12:05.277613 1 configmapobserver.go:84] configmaps "insights-config" not found I0424 19:12:05.300727 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0424 19:12:05.305508 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 19:12:05.305524 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0424 19:12:05.312941 1 secretconfigobserver.go:119] support secret does not exist I0424 19:12:05.315676 1 controller.go:554] The operator is healthy I0424 19:14:05.317795 1 controller.go:554] The operator is healthy I0424 19:16:05.316596 1 controller.go:554] The operator is healthy I0424 19:17:05.313945 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0424 19:17:05.318864 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 19:17:05.318882 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0424 19:17:05.323162 1 secretconfigobserver.go:119] support secret does not exist I0424 19:18:05.320160 1 controller.go:554] The operator is healthy I0424 19:19:25.346973 1 insightsuploader.go:146] Nothing to report since 2026-04-24T19:03:35Z I0424 19:20:05.317914 1 controller.go:554] The operator is healthy