I0425 02:57:49.805711 1 cmd.go:253] Using service-serving-cert provided certificates I0425 02:57:49.806129 1 observer_polling.go:159] Starting file observer I0425 02:57:50.432481 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0425 02:57:50.432684 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0425 02:57:50.434825 1 secure_serving.go:57] Forcing use of http/1.1 only W0425 02:57:50.434851 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0425 02:57:50.434856 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0425 02:57:50.434859 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0425 02:57:50.434862 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0425 02:57:50.434864 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0425 02:57:50.434867 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0425 02:57:50.434879 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0425 02:57:50.439992 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0425 02:57:50.440008 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"c57c5d9a-b8cb-4785-804c-9471c3c4dd9d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0425 02:57:50.442281 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0425 02:57:50.442296 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0425 02:57:50.442293 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0425 02:57:50.442295 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0425 02:57:50.442308 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0425 02:57:50.442311 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0425 02:57:50.442603 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0425 02:57:50.442853 1 secure_serving.go:213] Serving securely on [::]:8443 I0425 02:57:50.442875 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0425 02:57:50.447597 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0425 02:57:50.447626 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0425 02:57:50.447732 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0425 02:57:50.454831 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0425 02:57:50.454854 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0425 02:57:50.470321 1 secretconfigobserver.go:119] support secret does not exist I0425 02:57:50.476901 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0425 02:57:50.483709 1 secretconfigobserver.go:119] support secret does not exist I0425 02:57:50.491330 1 recorder.go:176] Pruning old reports every 8h14m26s, max age is 288h0m0s I0425 02:57:50.497820 1 periodic.go:216] Running clusterconfig gatherer I0425 02:57:50.497825 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0425 02:57:50.497843 1 insightsreport.go:296] Starting report retriever I0425 02:57:50.497852 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0425 02:57:50.497822 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0425 02:57:50.497867 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0425 02:57:50.497873 1 tasks_processing.go:45] number of workers: 64 I0425 02:57:50.498156 1 tasks_processing.go:69] worker 22 listening for tasks. I0425 02:57:50.498177 1 tasks_processing.go:69] worker 10 listening for tasks. I0425 02:57:50.498184 1 tasks_processing.go:71] worker 22 working on container_runtime_configs task. I0425 02:57:50.498173 1 tasks_processing.go:69] worker 27 listening for tasks. I0425 02:57:50.498191 1 tasks_processing.go:69] worker 26 listening for tasks. I0425 02:57:50.498195 1 tasks_processing.go:69] worker 28 listening for tasks. I0425 02:57:50.498210 1 tasks_processing.go:69] worker 47 listening for tasks. I0425 02:57:50.498238 1 tasks_processing.go:69] worker 63 listening for tasks. I0425 02:57:50.498256 1 tasks_processing.go:69] worker 29 listening for tasks. I0425 02:57:50.498293 1 tasks_processing.go:69] worker 23 listening for tasks. I0425 02:57:50.498320 1 tasks_processing.go:69] worker 34 listening for tasks. I0425 02:57:50.498327 1 tasks_processing.go:69] worker 0 listening for tasks. I0425 02:57:50.498331 1 tasks_processing.go:69] worker 48 listening for tasks. I0425 02:57:50.498334 1 tasks_processing.go:69] worker 35 listening for tasks. I0425 02:57:50.498340 1 tasks_processing.go:69] worker 9 listening for tasks. I0425 02:57:50.498341 1 tasks_processing.go:69] worker 49 listening for tasks. I0425 02:57:50.498344 1 tasks_processing.go:69] worker 36 listening for tasks. I0425 02:57:50.498363 1 tasks_processing.go:69] worker 32 listening for tasks. I0425 02:57:50.498373 1 tasks_processing.go:69] worker 8 listening for tasks. I0425 02:57:50.498174 1 tasks_processing.go:69] worker 33 listening for tasks. I0425 02:57:50.498456 1 tasks_processing.go:69] worker 6 listening for tasks. I0425 02:57:50.498493 1 tasks_processing.go:69] worker 11 listening for tasks. I0425 02:57:50.499239 1 tasks_processing.go:69] worker 12 listening for tasks. I0425 02:57:50.499261 1 tasks_processing.go:69] worker 31 listening for tasks. I0425 02:57:50.499528 1 tasks_processing.go:69] worker 50 listening for tasks. I0425 02:57:50.499533 1 tasks_processing.go:69] worker 59 listening for tasks. I0425 02:57:50.499547 1 tasks_processing.go:69] worker 2 listening for tasks. I0425 02:57:50.499544 1 tasks_processing.go:69] worker 54 listening for tasks. I0425 02:57:50.499551 1 tasks_processing.go:69] worker 13 listening for tasks. I0425 02:57:50.499558 1 tasks_processing.go:69] worker 55 listening for tasks. I0425 02:57:50.499551 1 tasks_processing.go:69] worker 57 listening for tasks. I0425 02:57:50.499565 1 tasks_processing.go:69] worker 14 listening for tasks. I0425 02:57:50.499562 1 tasks_processing.go:69] worker 60 listening for tasks. I0425 02:57:50.499559 1 tasks_processing.go:69] worker 58 listening for tasks. I0425 02:57:50.499572 1 tasks_processing.go:69] worker 51 listening for tasks. I0425 02:57:50.499573 1 tasks_processing.go:69] worker 15 listening for tasks. I0425 02:57:50.499574 1 tasks_processing.go:69] worker 4 listening for tasks. I0425 02:57:50.499580 1 tasks_processing.go:69] worker 52 listening for tasks. I0425 02:57:50.499583 1 tasks_processing.go:69] worker 19 listening for tasks. I0425 02:57:50.499585 1 tasks_processing.go:69] worker 38 listening for tasks. I0425 02:57:50.499589 1 tasks_processing.go:69] worker 3 listening for tasks. I0425 02:57:50.499614 1 tasks_processing.go:69] worker 20 listening for tasks. I0425 02:57:50.499607 1 tasks_processing.go:69] worker 62 listening for tasks. I0425 02:57:50.499623 1 tasks_processing.go:69] worker 30 listening for tasks. I0425 02:57:50.499645 1 tasks_processing.go:69] worker 53 listening for tasks. I0425 02:57:50.499651 1 tasks_processing.go:69] worker 1 listening for tasks. I0425 02:57:50.499665 1 tasks_processing.go:69] worker 39 listening for tasks. I0425 02:57:50.499674 1 tasks_processing.go:69] worker 61 listening for tasks. I0425 02:57:50.499675 1 tasks_processing.go:69] worker 45 listening for tasks. I0425 02:57:50.499684 1 tasks_processing.go:69] worker 37 listening for tasks. I0425 02:57:50.499694 1 tasks_processing.go:69] worker 21 listening for tasks. I0425 02:57:50.499703 1 tasks_processing.go:69] worker 56 listening for tasks. I0425 02:57:50.499717 1 tasks_processing.go:69] worker 40 listening for tasks. I0425 02:57:50.499718 1 tasks_processing.go:69] worker 44 listening for tasks. I0425 02:57:50.499720 1 tasks_processing.go:69] worker 17 listening for tasks. I0425 02:57:50.499717 1 tasks_processing.go:69] worker 46 listening for tasks. I0425 02:57:50.499726 1 tasks_processing.go:71] worker 31 working on certificate_signing_requests task. I0425 02:57:50.499728 1 tasks_processing.go:71] worker 51 working on openstack_version task. I0425 02:57:50.499731 1 tasks_processing.go:69] worker 42 listening for tasks. I0425 02:57:50.499735 1 tasks_processing.go:71] worker 14 working on crds task. I0425 02:57:50.499740 1 tasks_processing.go:69] worker 18 listening for tasks. I0425 02:57:50.499733 1 tasks_processing.go:71] worker 13 working on jaegers task. I0425 02:57:50.499740 1 tasks_processing.go:71] worker 48 working on ingress task. I0425 02:57:50.499745 1 tasks_processing.go:69] worker 41 listening for tasks. I0425 02:57:50.499749 1 tasks_processing.go:71] worker 27 working on overlapping_namespace_uids task. I0425 02:57:50.499756 1 tasks_processing.go:71] worker 63 working on active_alerts task. I0425 02:57:50.499762 1 tasks_processing.go:69] worker 43 listening for tasks. I0425 02:57:50.499770 1 tasks_processing.go:71] worker 32 working on image_pruners task. I0425 02:57:50.499843 1 tasks_processing.go:69] worker 16 listening for tasks. I0425 02:57:50.499855 1 tasks_processing.go:71] worker 55 working on machines task. I0425 02:57:50.499863 1 tasks_processing.go:71] worker 58 working on infrastructures task. I0425 02:57:50.499865 1 tasks_processing.go:71] worker 59 working on olm_operators task. I0425 02:57:50.499872 1 tasks_processing.go:71] worker 28 working on machine_configs task. I0425 02:57:50.499744 1 tasks_processing.go:71] worker 10 working on version task. I0425 02:57:50.499878 1 tasks_processing.go:69] worker 24 listening for tasks. I0425 02:57:50.499859 1 tasks_processing.go:71] worker 50 working on proxies task. I0425 02:57:50.499902 1 tasks_processing.go:71] worker 23 working on metrics task. I0425 02:57:50.499858 1 tasks_processing.go:71] worker 60 working on authentication task. I0425 02:57:50.500041 1 tasks_processing.go:71] worker 45 working on sap_config task. I0425 02:57:50.499866 1 tasks_processing.go:71] worker 29 working on node_features task. I0425 02:57:50.500061 1 tasks_processing.go:71] worker 26 working on machine_sets task. I0425 02:57:50.499999 1 tasks_processing.go:71] worker 47 working on nodes task. I0425 02:57:50.500016 1 tasks_processing.go:71] worker 9 working on networks task. I0425 02:57:50.500111 1 tasks_processing.go:69] worker 7 listening for tasks. I0425 02:57:50.500142 1 tasks_processing.go:71] worker 37 working on container_images task. I0425 02:57:50.500148 1 tasks_processing.go:71] worker 7 working on machine_config_pools task. I0425 02:57:50.500178 1 tasks_processing.go:71] worker 21 working on openshift_machine_api_events task. I0425 02:57:50.500398 1 tasks_processing.go:71] worker 8 working on sap_datahubs task. I0425 02:57:50.500445 1 tasks_processing.go:71] worker 6 working on image_registries task. I0425 02:57:50.500448 1 tasks_processing.go:71] worker 44 working on sap_pods task. I0425 02:57:50.500473 1 tasks_processing.go:71] worker 2 working on lokistack task. I0425 02:57:50.500480 1 tasks_processing.go:71] worker 46 working on machine_autoscalers task. I0425 02:57:50.499906 1 tasks_processing.go:69] worker 25 listening for tasks. I0425 02:57:50.500484 1 tasks_processing.go:71] worker 11 working on pod_network_connectivity_checks task. I0425 02:57:50.500493 1 tasks_processing.go:71] worker 25 working on support_secret task. I0425 02:57:50.500494 1 tasks_processing.go:71] worker 41 working on storage_cluster task. I0425 02:57:50.500502 1 tasks_processing.go:71] worker 16 working on cluster_apiserver task. I0425 02:57:50.500510 1 tasks_processing.go:71] worker 42 working on cost_management_metrics_configs task. I0425 02:57:50.500015 1 tasks_processing.go:71] worker 34 working on schedulers task. I0425 02:57:50.500535 1 tasks_processing.go:71] worker 24 working on openshift_logging task. I0425 02:57:50.500447 1 tasks_processing.go:71] worker 54 working on aggregated_monitoring_cr_names task. I0425 02:57:50.500543 1 tasks_processing.go:71] worker 15 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0425 02:57:50.500566 1 tasks_processing.go:71] worker 52 working on openstack_controlplanes task. I0425 02:57:50.500619 1 tasks_processing.go:71] worker 39 working on tsdb_status task. I0425 02:57:50.500538 1 tasks_processing.go:71] worker 3 working on dvo_metrics task. I0425 02:57:50.500475 1 tasks_processing.go:71] worker 17 working on install_plans task. I0425 02:57:50.500028 1 tasks_processing.go:71] worker 56 working on operators_pods_and_events task. I0425 02:57:50.500023 1 tasks_processing.go:71] worker 0 working on validating_webhook_configurations task. I0425 02:57:50.500027 1 tasks_processing.go:71] worker 36 working on operators task. I0425 02:57:50.500034 1 tasks_processing.go:71] worker 61 working on monitoring_persistent_volumes task. I0425 02:57:50.500010 1 tasks_processing.go:71] worker 35 working on pdbs task. I0425 02:57:50.500443 1 tasks_processing.go:71] worker 40 working on oauths task. I0425 02:57:50.500571 1 tasks_processing.go:71] worker 18 working on service_accounts task. I0425 02:57:50.500586 1 tasks_processing.go:71] worker 30 working on nodenetworkconfigurationpolicies task. I0425 02:57:50.500022 1 tasks_processing.go:71] worker 49 working on machine_healthchecks task. I0425 02:57:50.500591 1 tasks_processing.go:71] worker 20 working on storage_classes task. I0425 02:57:50.500588 1 tasks_processing.go:71] worker 53 working on openstack_dataplanenodesets task. I0425 02:57:50.500589 1 tasks_processing.go:71] worker 1 working on openstack_dataplanedeployments task. I0425 02:57:50.500600 1 tasks_processing.go:71] worker 12 working on nodenetworkstates task. I0425 02:57:50.500605 1 tasks_processing.go:71] worker 33 working on mutating_webhook_configurations task. I0425 02:57:50.499860 1 tasks_processing.go:71] worker 57 working on ceph_cluster task. I0425 02:57:50.500610 1 tasks_processing.go:71] worker 43 working on silenced_alerts task. I0425 02:57:50.500610 1 tasks_processing.go:71] worker 4 working on node_logs task. I0425 02:57:50.500612 1 tasks_processing.go:71] worker 62 working on feature_gates task. I0425 02:57:50.500614 1 tasks_processing.go:71] worker 19 working on image task. I0425 02:57:50.500618 1 tasks_processing.go:71] worker 38 working on ingress_certificates task. I0425 02:57:50.502800 1 tasks_processing.go:71] worker 22 working on qemu_kubevirt_launcher_logs task. I0425 02:57:50.502810 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 4.602994ms to process 0 records I0425 02:57:50.500652 1 tasks_processing.go:69] worker 5 listening for tasks. I0425 02:57:50.502893 1 tasks_processing.go:71] worker 5 working on config_maps task. E0425 02:57:50.506089 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.139.217:9091: connect: connection refused I0425 02:57:50.506129 1 tasks_processing.go:71] worker 39 working on clusterroles task. E0425 02:57:50.506129 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.139.217:9091: connect: connection refused E0425 02:57:50.506142 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.139.217:9091: connect: connection refused I0425 02:57:50.506161 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 5.334684ms to process 0 records E0425 02:57:50.506173 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.139.217:9091: connect: connection refused I0425 02:57:50.506191 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 6.108724ms to process 0 records I0425 02:57:50.506205 1 tasks_processing.go:74] worker 23 stopped. I0425 02:57:50.511173 1 tasks_processing.go:74] worker 26 stopped. I0425 02:57:50.511188 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 11.101011ms to process 0 records E0425 02:57:50.511388 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0425 02:57:50.511413 1 tasks_processing.go:74] worker 29 stopped. I0425 02:57:50.511424 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 11.351813ms to process 0 records I0425 02:57:50.511483 1 tasks_processing.go:74] worker 50 stopped. I0425 02:57:50.511627 1 recorder.go:75] Recording config/proxy with fingerprint=06eb1eef1f09eae5c5a034d14a68d5beb3a73af89eff80bfb372964934b3f2c5 I0425 02:57:50.511648 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 11.585463ms to process 1 records E0425 02:57:50.511667 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0425 02:57:50.511677 1 gather.go:177] gatherer "clusterconfig" function "machines" took 11.652601ms to process 0 records I0425 02:57:50.511680 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0425 02:57:50.511693 1 controller.go:368] The initial operator extension status is healthy I0425 02:57:50.511705 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0425 02:57:50.511712 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0425 02:57:50.511716 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0425 02:57:50.511732 1 tasks_processing.go:74] worker 55 stopped. I0425 02:57:50.511738 1 controller.go:531] The operator is still being initialized I0425 02:57:50.511744 1 controller.go:554] The operator is healthy I0425 02:57:50.511802 1 tasks_processing.go:74] worker 48 stopped. I0425 02:57:50.511821 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0425 02:57:50.511896 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0425 02:57:50.511906 1 recorder.go:75] Recording config/ingress with fingerprint=39a970e1c0231ab4f122fd7668dfdec67974e239d3bb43bf8575a7d96facfe87 I0425 02:57:50.511917 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 11.932118ms to process 1 records W0425 02:57:50.511964 1 operator.go:328] started I0425 02:57:50.511989 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0425 02:57:50.512043 1 tasks_processing.go:74] worker 32 stopped. I0425 02:57:50.512159 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=7adaf41275bd9a6ba7fffa77c26849d9c1c42760954bbca764fb4a2df1770b52 I0425 02:57:50.512172 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 11.980498ms to process 1 records I0425 02:57:50.512251 1 tasks_processing.go:74] worker 60 stopped. I0425 02:57:50.512309 1 recorder.go:75] Recording config/authentication with fingerprint=47ee5a5d904410fbb33e3ccf7a2af0722012adbe1256513fb3a6f9f4e4f4f97f I0425 02:57:50.512318 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 11.767514ms to process 1 records I0425 02:57:50.512390 1 tasks_processing.go:74] worker 9 stopped. I0425 02:57:50.512412 1 recorder.go:75] Recording config/network with fingerprint=34b1865c8661c933277304d8c1742b25820a849164d7ad269b7718d3189d2826 I0425 02:57:50.512419 1 gather.go:177] gatherer "clusterconfig" function "networks" took 11.696391ms to process 1 records I0425 02:57:50.512506 1 tasks_processing.go:74] worker 40 stopped. I0425 02:57:50.512576 1 recorder.go:75] Recording config/oauth with fingerprint=8df9a5a75ecbd881f402ba1dd51ac3c4c123a41ebf90d00ab14aeee88cdb9c8f I0425 02:57:50.512583 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 11.250458ms to process 1 records I0425 02:57:50.517051 1 tasks_processing.go:74] worker 7 stopped. I0425 02:57:50.517071 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 16.889837ms to process 0 records I0425 02:57:50.517428 1 tasks_processing.go:74] worker 46 stopped. I0425 02:57:50.517442 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 16.932666ms to process 0 records I0425 02:57:50.520528 1 tasks_processing.go:74] worker 11 stopped. E0425 02:57:50.520546 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0425 02:57:50.520557 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 20.021567ms to process 0 records I0425 02:57:50.521241 1 tasks_processing.go:74] worker 24 stopped. I0425 02:57:50.521260 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 20.692279ms to process 0 records I0425 02:57:50.521337 1 tasks_processing.go:74] worker 8 stopped. I0425 02:57:50.521351 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 20.880058ms to process 0 records I0425 02:57:50.521367 1 tasks_processing.go:74] worker 44 stopped. I0425 02:57:50.521379 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 20.895517ms to process 0 records I0425 02:57:50.521422 1 tasks_processing.go:74] worker 42 stopped. I0425 02:57:50.521434 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 20.900013ms to process 0 records I0425 02:57:50.522069 1 tasks_processing.go:74] worker 53 stopped. I0425 02:57:50.522079 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 20.222732ms to process 0 records I0425 02:57:50.522325 1 tasks_processing.go:74] worker 12 stopped. I0425 02:57:50.522337 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 20.315876ms to process 0 records I0425 02:57:50.522422 1 tasks_processing.go:74] worker 30 stopped. I0425 02:57:50.522434 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 20.960958ms to process 0 records I0425 02:57:50.522747 1 tasks_processing.go:74] worker 41 stopped. I0425 02:57:50.522758 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 22.23963ms to process 0 records I0425 02:57:50.522895 1 tasks_processing.go:74] worker 1 stopped. I0425 02:57:50.522911 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 20.910797ms to process 0 records I0425 02:57:50.523983 1 tasks_processing.go:74] worker 13 stopped. I0425 02:57:50.523993 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 24.230346ms to process 0 records I0425 02:57:50.524262 1 tasks_processing.go:74] worker 2 stopped. I0425 02:57:50.524273 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 23.774974ms to process 0 records I0425 02:57:50.525326 1 tasks_processing.go:74] worker 51 stopped. I0425 02:57:50.525339 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 25.587236ms to process 0 records I0425 02:57:50.527540 1 tasks_processing.go:74] worker 45 stopped. I0425 02:57:50.527552 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 27.485853ms to process 0 records I0425 02:57:50.528516 1 tasks_processing.go:74] worker 52 stopped. I0425 02:57:50.528526 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 27.889151ms to process 0 records I0425 02:57:50.529052 1 tasks_processing.go:74] worker 28 stopped. E0425 02:57:50.529064 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0425 02:57:50.529095 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0425 02:57:50.529140 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 29.166504ms to process 1 records I0425 02:57:50.529386 1 tasks_processing.go:74] worker 57 stopped. I0425 02:57:50.529393 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 27.194857ms to process 0 records I0425 02:57:50.529523 1 tasks_processing.go:74] worker 16 stopped. I0425 02:57:50.529569 1 recorder.go:75] Recording config/apiserver with fingerprint=bba5ec440c1e97bff1ba561c8a6dfb48bdcc716174ca09929c38db4713c4de2c I0425 02:57:50.529583 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 28.875349ms to process 1 records I0425 02:57:50.530883 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0425 02:57:50.530895 1 controller.go:254] Source scaController *sca.Controller is not ready I0425 02:57:50.530899 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0425 02:57:50.530902 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0425 02:57:50.530904 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0425 02:57:50.530921 1 controller.go:531] The operator is still being initialized I0425 02:57:50.530927 1 controller.go:554] The operator is healthy I0425 02:57:50.531251 1 tasks_processing.go:74] worker 14 stopped. I0425 02:57:50.531753 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=f1a1bbf9a07d2e5c79b5e1ffcb6d67e35d424a9db504ca302baa70d8261f313d I0425 02:57:50.531957 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=028cfbe60794f93c9c61cc19ff453dd98498540c7dee2e86f465e644939f8e27 I0425 02:57:50.531966 1 gather.go:177] gatherer "clusterconfig" function "crds" took 31.503191ms to process 2 records I0425 02:57:50.532039 1 recorder.go:75] Recording config/image with fingerprint=e2878bddcf82dd522128b638f49a99fff465186eb8f050de26f43a9d73805f1f I0425 02:57:50.532048 1 gather.go:177] gatherer "clusterconfig" function "image" took 28.831053ms to process 1 records I0425 02:57:50.532052 1 tasks_processing.go:74] worker 19 stopped. I0425 02:57:50.532128 1 tasks_processing.go:74] worker 58 stopped. I0425 02:57:50.532538 1 recorder.go:75] Recording config/infrastructure with fingerprint=6b8a6ad4a8161968519c30c819cd79547048fcd1ae565c79093a73dafca33788 I0425 02:57:50.532549 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 31.667889ms to process 1 records I0425 02:57:50.533274 1 tasks_processing.go:74] worker 25 stopped. E0425 02:57:50.533292 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0425 02:57:50.533306 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 32.769804ms to process 0 records I0425 02:57:50.533889 1 tasks_processing.go:74] worker 49 stopped. E0425 02:57:50.533902 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0425 02:57:50.533911 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 32.181738ms to process 0 records I0425 02:57:50.534401 1 tasks_processing.go:74] worker 20 stopped. I0425 02:57:50.534500 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=a4bc441b56b28f4a4724b4ca54649093f648d989d7939c7fbf51d951b939afa8 I0425 02:57:50.534528 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=58d85b242a5e0f4a60069550317e2a909aacd6a61f3dbe266050a2f7ff613082 I0425 02:57:50.534536 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 32.600217ms to process 2 records I0425 02:57:50.536022 1 tasks_processing.go:74] worker 47 stopped. I0425 02:57:50.536350 1 recorder.go:75] Recording config/node/ip-10-0-0-192.ec2.internal with fingerprint=7e8e41bcd797ce5b065599108ff27fc06cd5dcd73ac98ebd72e1030975a48a2c I0425 02:57:50.536416 1 recorder.go:75] Recording config/node/ip-10-0-1-179.ec2.internal with fingerprint=79e6c1f0594a103d6ffd08bc542bda3cf07121fe155c99bdb0bd07e77d4e51ba I0425 02:57:50.536484 1 recorder.go:75] Recording config/node/ip-10-0-2-178.ec2.internal with fingerprint=4b6e8bdb8ef2855260053604d89e047176999da45299c04a4ea7cfefef709636 I0425 02:57:50.536498 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 35.930045ms to process 3 records I0425 02:57:50.536931 1 tasks_processing.go:74] worker 33 stopped. I0425 02:57:50.537237 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=5eead6de6d08213cfc4059503961173a685d7f9a0b106f618c7b7f1c8625c776 I0425 02:57:50.537294 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=b39ae807d0518a38815e90236cf79a6a32bcde7ebac395a8fc21ab45e1129ce9 I0425 02:57:50.537346 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=80dbffd43d3c6cbe99320ca501de5dfa5ba156e00674ec896980398da3139323 I0425 02:57:50.537362 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 34.1208ms to process 3 records I0425 02:57:50.537372 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 33.730429ms to process 0 records I0425 02:57:50.537381 1 tasks_processing.go:74] worker 4 stopped. I0425 02:57:50.537532 1 tasks_processing.go:74] worker 6 stopped. I0425 02:57:50.538084 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=7257b171403f7f5d7cfab45ce69d8ac67895186d15868722f3902295f956780f I0425 02:57:50.538134 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 37.067661ms to process 1 records I0425 02:57:50.539084 1 tasks_processing.go:74] worker 61 stopped. I0425 02:57:50.539136 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 38.096744ms to process 0 records I0425 02:57:50.539214 1 tasks_processing.go:74] worker 21 stopped. I0425 02:57:50.539232 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 39.025776ms to process 0 records I0425 02:57:50.539551 1 tasks_processing.go:74] worker 0 stopped. I0425 02:57:50.539657 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=50360774e66989169b95e50028adca16800cf43135cf15f2ebdf26f2cea307f7 I0425 02:57:50.539703 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=a64762e3268136adddf30f14dd91acd73ffd45f66f69119c40334302f4c41920 I0425 02:57:50.539752 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=6c041c8a14eb8b91c9d7a741a5eaf21c255c3bd027b4f744eb841984b7387b0f I0425 02:57:50.539778 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=1bad73b232a95815b3aaac9495f7d828db85612eae49721d318a4b628b385a5c I0425 02:57:50.539806 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=e7cd352dfcc1f7d679063004114648356ef85a726310c98639ee700bd42e4624 I0425 02:57:50.539830 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=2a7af10930caa9ca8d523f460c638d05b39cdb0f6f3d3951d010f579644f2d3a I0425 02:57:50.539854 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=a2cfe310659f0cecd4da74eb05e68ee6efc9a0e2f79e030f6283a23124143dc5 I0425 02:57:50.539875 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=42dc5d059f6bb7228043d29e4d7fca0a100e5af8883645031c89935b451c4c23 I0425 02:57:50.539906 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=1d555e0fc3f3b99972354ddc702f2d981a98d454582ae526b54db0e9f20c53c6 I0425 02:57:50.539937 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=d7ed1720f6a7f0b236b03b21393a984fbab66fd679467e2a368a3b8f73a00270 I0425 02:57:50.539959 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=5ad7561973a04a769e50496937000988c865495ad4b713a4275bd15b0e3de68a I0425 02:57:50.539982 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=beb63be59582d7bb2593e6d25dc7f7b01be69eb40349e9268defac7460156500 I0425 02:57:50.540038 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=8238c0a09ae3b6e5a8a8add57581e86e2ba0d566df06b81381878a4671e3eabc I0425 02:57:50.540047 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 38.609429ms to process 13 records I0425 02:57:50.540575 1 tasks_processing.go:74] worker 31 stopped. I0425 02:57:50.540593 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 40.835337ms to process 0 records I0425 02:57:50.542353 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0425 02:57:50.542371 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0425 02:57:50.542354 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0425 02:57:50.544391 1 tasks_processing.go:74] worker 34 stopped. I0425 02:57:50.544493 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=ca4151bd01795b9439cd4d6cec1cfbdb91bce226391b2706ab6b5c196fbac296 I0425 02:57:50.544512 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 43.851553ms to process 1 records I0425 02:57:50.544596 1 tasks_processing.go:74] worker 35 stopped. I0425 02:57:50.544628 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=69f9b4fd2d3fc905c6a1efd6f971d3a9458700659aaea6b0ba45e0d71e0bd72a I0425 02:57:50.544651 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=9cfffe8eca84c1df53ad8f2de2428f4b81d672ce75d43424dc406682b02baf1a I0425 02:57:50.544669 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=96131edaddd4469f3a8c31e5e54f914988d83c34ff2a1e809720e0cf2f82bce5 I0425 02:57:50.545139 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=090299d92dc703fd2faca8ffa10d90b40cced819ddc9f78d81f97029b07973aa I0425 02:57:50.545239 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=13f01e987fefc58f3e09d6cd96806a3d2bd48769fc415ba10c364d33055c6ca7 I0425 02:57:50.545306 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=3c640236c19978583729511d238b00db05f90762199422dced79a3d21c5a64dd I0425 02:57:50.545347 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=85c1135973d4fc8281a46aeab355cab17000396cb84258882fac7a32b51c0095 I0425 02:57:50.545373 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=15eeb260539af1a2e4cb6f63fa19164dd67176b9a6c8413b82dca1c718be9487 I0425 02:57:50.545405 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=49ea32f9e05a8ef56a95b57845a07ac4afdf3594c7ebf12f38eb75ea7ce5017c I0425 02:57:50.545436 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=faac509bf28ea9c803da5482429f6d3e953d3936fa752910d0967e1e17eb7b3e I0425 02:57:50.545263 1 gather_logs.go:145] no pods in namespace were found I0425 02:57:50.545471 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=37cca48d3624e83b3e19ba8e4e16fd650aed56fd1604dc9648bde26d77d6d150 I0425 02:57:50.545484 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 43.389362ms to process 11 records I0425 02:57:50.545494 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 42.648496ms to process 0 records I0425 02:57:50.545512 1 tasks_processing.go:74] worker 22 stopped. W0425 02:57:50.545999 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0425 02:57:50.547865 1 base_controller.go:82] Caches are synced for ConfigController I0425 02:57:50.547884 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0425 02:57:50.549060 1 tasks_processing.go:74] worker 59 stopped. I0425 02:57:50.549177 1 recorder.go:75] Recording config/olm_operators with fingerprint=16b3928133f9b36f98d5fd7e0198d232c660254163cb2d80e16e8b63104b7299 I0425 02:57:50.549203 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 49.17476ms to process 1 records I0425 02:57:50.549223 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 48.155163ms to process 0 records I0425 02:57:50.549246 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0425 02:57:50.549264 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 49.337616ms to process 1 records I0425 02:57:50.549275 1 tasks_processing.go:74] worker 27 stopped. I0425 02:57:50.549288 1 tasks_processing.go:74] worker 56 stopped. I0425 02:57:50.551102 1 tasks_processing.go:74] worker 10 stopped. I0425 02:57:50.551358 1 recorder.go:75] Recording config/version with fingerprint=2fe401d34f1fa431afcadf08bd350ede69f39f47f4e02cbc7054c809ed2ce3ea I0425 02:57:50.551372 1 recorder.go:75] Recording config/id with fingerprint=5fa784f36840890777d9ed0a0dc5c4160e0e9bf363390d8faef2aae1b139c4da I0425 02:57:50.551379 1 gather.go:177] gatherer "clusterconfig" function "version" took 51.208496ms to process 2 records I0425 02:57:50.551698 1 tasks_processing.go:74] worker 39 stopped. I0425 02:57:50.551899 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=a552c7659bfed38c63fc42bd8038c0cdf6193151a9cc635245ce0e450732e129 I0425 02:57:50.552004 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=636ed36d9b8258b9ba63ca242f4f7be675d0b1485e9562be13df9cdede26bd33 I0425 02:57:50.552014 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 45.559876ms to process 2 records I0425 02:57:50.552460 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0425 02:57:50.555631 1 tasks_processing.go:74] worker 62 stopped. I0425 02:57:50.555790 1 recorder.go:75] Recording config/featuregate with fingerprint=fa711fabd3c29849bf6fa8035d2a665c22f9d0f52dc9cf3e287f6c3c5a3692bd I0425 02:57:50.555811 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 52.941651ms to process 1 records I0425 02:57:50.558343 1 configmapobserver.go:84] configmaps "insights-config" not found I0425 02:57:50.563312 1 tasks_processing.go:74] worker 54 stopped. I0425 02:57:50.563337 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 62.754244ms to process 0 records I0425 02:57:50.568597 1 tasks_processing.go:74] worker 37 stopped. I0425 02:57:50.570082 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-66dbbb6468-mbhkb with fingerprint=1d8d7511d2f7b45cf8c37d581881ab1c6af99aaf4bbf250588bfb938aa759792 I0425 02:57:50.570209 1 recorder.go:75] Recording config/running_containers with fingerprint=bc3a4a0dc9f9c82f5a917a626f05d23c419b0b55af553c2416bd889ed37f56f6 I0425 02:57:50.570223 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 68.439045ms to process 2 records I0425 02:57:50.577233 1 tasks_processing.go:74] worker 15 stopped. I0425 02:57:50.577252 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 76.672185ms to process 0 records I0425 02:57:50.582167 1 tasks_processing.go:74] worker 38 stopped. E0425 02:57:50.582190 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0425 02:57:50.582201 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2pstnblpfvopedep638n6fq8493r2vv0-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2pstnblpfvopedep638n6fq8493r2vv0-primary-cert-bundle-secret" not found I0425 02:57:50.582259 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=64a9ee72ca46e931b07da98c90fecfe9ad6b134670dd6418847094558d362cf0 I0425 02:57:50.582274 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 79.363343ms to process 1 records I0425 02:57:50.583542 1 tasks_processing.go:74] worker 5 stopped. E0425 02:57:50.583557 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0425 02:57:50.583562 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0425 02:57:50.583566 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0425 02:57:50.583576 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0425 02:57:50.583636 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=2b7e7d968c1ca0e033f2ae1514829fefa6008cf7d89addbe266d351cfb9cca19 I0425 02:57:50.583644 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0425 02:57:50.583649 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0425 02:57:50.583660 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0425 02:57:50.583698 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0425 02:57:50.583706 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0425 02:57:50.583711 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 80.621879ms to process 7 records I0425 02:57:50.609160 1 cluster_transfer.go:102] no available accepted cluster transfer I0425 02:57:50.609180 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0425 02:57:50.612370 1 base_controller.go:82] Caches are synced for LoggingSyncer I0425 02:57:50.612385 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0425 02:57:50.630842 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0425 02:57:50.639082 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0425 02:57:50.790615 1 sca.go:266] etc-pki-entitlement secret successfully created I0425 02:57:50.790635 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated W0425 02:57:51.546096 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0425 02:57:51.575323 1 tasks_processing.go:74] worker 63 stopped. I0425 02:57:51.575368 1 recorder.go:75] Recording config/alerts with fingerprint=87aeb8d9f9c70afabe2437669e00fe0a78d0cb373ac7dd0ebcfbd5468a429799 I0425 02:57:51.575384 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.075548727s to process 1 records I0425 02:57:51.575656 1 tasks_processing.go:74] worker 43 stopped. I0425 02:57:51.575678 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0425 02:57:51.575690 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.0733072s to process 1 records W0425 02:57:52.545647 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0425 02:57:52.974358 1 tasks_processing.go:74] worker 36 stopped. I0425 02:57:52.974485 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=cedbffb00d1a1dbabee339471434a11f94bb7cabc809f014b7f869b40c973423 I0425 02:57:52.974552 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0425 02:57:52.974583 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=2a4d64778bedd32c401982460851495474134d804fd9ef6138466a6b4eda4529 I0425 02:57:52.974593 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0425 02:57:52.974619 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=6a591d81d708aac65f99f35738846feac9542b31ac370f5abedd15f9548d0f1b I0425 02:57:52.974635 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0425 02:57:52.974658 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=aead468f76a469ee8c6527830e2fb561b6215a40dbc8f7bef8b7df174f3df443 I0425 02:57:52.974685 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=bf52c26396dfaee1f41e5288b86721207c00be47ccb1a51f6acb43b570c5db53 I0425 02:57:52.974708 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=9008092ad1c028703b6fdce53b024c6fe1b9fc6c9834077ebd6854679adf3e0f I0425 02:57:52.974741 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=a5a77fdbbbd2e9095673bf0575e6f30853b8b6a149ea23e4c9de2a4841cbc29c I0425 02:57:52.975015 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=bf77643026a4a8b3d17e281c0887018dcfe8985e33217aac9d5e0aea51749902 I0425 02:57:52.975035 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=cbed40e519244956a5569fc128b2801a31c15332859fecff0edabbb01660345a I0425 02:57:52.975043 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0425 02:57:52.975059 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=749de4907b26ce623bf906c050e0e2e70d9611dcf0d2abf143220837502f580b I0425 02:57:52.975068 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0425 02:57:52.975086 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=1453ff650b041a3d442d35f7b7807a34d476cb96130f508774d84e4c60707c21 I0425 02:57:52.975095 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0425 02:57:52.975130 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=6a878fe8122577f83ce5b3ca351f3be6cbdb82515e184a2769b52dd1778e1b60 I0425 02:57:52.975178 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0425 02:57:52.975229 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=71c60d3298179278b8310672e8b3bd8781aae682d88a66b642ce05903041836f I0425 02:57:52.975335 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=ba044ecca6a8fe473e31911ff6aa87c7bcd7a3ef8c52ac303a4b046cae7399a6 I0425 02:57:52.975347 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0425 02:57:52.975354 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0425 02:57:52.975375 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0425 02:57:52.975399 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=2933fa3c9457104960c3e94b7f7006185869471541add6cf0f4ae76008a15995 I0425 02:57:52.975424 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=044d6047c9295fc91249c9429d5d17069e0ab57eb447c86b872f4e9fc69f8b3c I0425 02:57:52.975434 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0425 02:57:52.975449 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=25dd3dc3543924b20f77cd3f6794304c09d022b98aa4b0cd8e8f7bbe63cf5d1e I0425 02:57:52.975459 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0425 02:57:52.975479 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=0d3d0dbd5c5bc6ca91d99c1241ce24e5e69acd5df72d7c278a759351e33a6081 I0425 02:57:52.975490 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0425 02:57:52.975509 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=4aa93f6f4dca33f9dbd385b588248eff3f2c2e0bce6f908b93177a819f6a6cad I0425 02:57:52.975532 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=0caab5e36e2755be702bccd0995e0dda928a3c7dbf0bf4cacf3a9283dc885d40 I0425 02:57:52.975551 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=3e51ce3e280071c0d710932a9b5e0a2b8baa8618287a6b6d145ff1203fc5a6d2 I0425 02:57:52.975577 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=2f3f2b168e9730bbcf3b681569c329b9b8c164de76d6a142f3e0d90bae7890ab I0425 02:57:52.975587 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0425 02:57:52.975610 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=db5780283777058da21eaf90ca25ffbcdc317e5a8a9aee92add0138b7bc2aa48 I0425 02:57:52.975629 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0425 02:57:52.975639 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0425 02:57:52.975646 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.473379105s to process 39 records W0425 02:57:53.545745 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0425 02:57:54.546058 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0425 02:57:55.543610 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0425 02:57:55.543640 1 tasks_processing.go:74] worker 3 stopped. E0425 02:57:55.543666 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0425 02:57:55.543689 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0425 02:57:55.543704 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.042846425s to process 1 records I0425 02:58:03.347751 1 tasks_processing.go:74] worker 17 stopped. I0425 02:58:03.347812 1 recorder.go:75] Recording config/installplans with fingerprint=95dfd5f33a9a46199239158a0ec0183b629818eca24ae41f4d706d7a3d604aec I0425 02:58:03.347830 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.846873066s to process 1 records I0425 02:58:04.110078 1 tasks_processing.go:74] worker 18 stopped. I0425 02:58:04.110429 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=796bd9f20e825fd32af435d21e60c10c3ec4910fa1211c5a6338f5b81c3f8071 I0425 02:58:04.110451 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.608967707s to process 1 records E0425 02:58:04.110524 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.612s with: function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_configs\" failed with an error, function \"support_secret\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error" I0425 02:58:04.111639 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "tsdb_status" failed with an error, function "metrics" failed with an error, function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_configs" failed with an error, function "support_secret" failed with an error, function "machine_healthchecks" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error I0425 02:58:04.111655 1 periodic.go:216] Running workloads gatherer I0425 02:58:04.111674 1 tasks_processing.go:45] number of workers: 2 I0425 02:58:04.111683 1 tasks_processing.go:69] worker 1 listening for tasks. I0425 02:58:04.111687 1 tasks_processing.go:71] worker 1 working on workload_info task. I0425 02:58:04.111699 1 tasks_processing.go:69] worker 0 listening for tasks. I0425 02:58:04.111714 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0425 02:58:04.120818 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-179.ec2.internal... I0425 02:58:04.120828 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-178.ec2.internal... I0425 02:58:04.120841 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-192.ec2.internal... I0425 02:58:04.150449 1 tasks_processing.go:74] worker 0 stopped. I0425 02:58:04.150471 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 38.716754ms to process 0 records I0425 02:58:08.529836 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 4s I0425 02:58:08.575417 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0425 02:58:08.600180 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (26ms) I0425 02:58:08.615348 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (15ms) I0425 02:58:08.629653 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (14ms) I0425 02:58:08.645747 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (16ms) I0425 02:58:08.668957 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (23ms) I0425 02:58:08.692353 1 gather_workloads_info.go:387] No image sha256:9d6d63dce784df5863e6a0ba4c11317a26c9d756d4e62d47759c97190a71cce5 (23ms) I0425 02:58:08.704389 1 gather_workloads_info.go:387] No image sha256:572ebf7c7fab1a518582f3ca46c866730423cce6e8a19f614dc15fd61bf0c65a (12ms) I0425 02:58:08.716434 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (12ms) I0425 02:58:08.731650 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (15ms) I0425 02:58:08.746568 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (15ms) I0425 02:58:08.764644 1 gather_workloads_info.go:387] No image sha256:5940cc34b322325618929b1e0eb3864dae73d54c8f5b474e218dfb28123a61c7 (18ms) I0425 02:58:08.789699 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (25ms) I0425 02:58:08.888469 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (99ms) I0425 02:58:08.986081 1 gather_workloads_info.go:387] No image sha256:0acfe677ed58031f8381cb8a4f2d8917ec4cfa95485e2a46529dffae54c75ace (98ms) I0425 02:58:09.088328 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (102ms) I0425 02:58:09.186051 1 gather_workloads_info.go:387] No image sha256:0b2e6536a0ca3afc5e64a218087a6287c74b77a359075a385f1949ca5bd51b22 (98ms) I0425 02:58:09.287140 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (101ms) I0425 02:58:09.385741 1 gather_workloads_info.go:387] No image sha256:9ef6dc311b880a6442b7a3a19006ddd955c7da36850bf000f5c9b68fd0862dd4 (99ms) I0425 02:58:09.486952 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (101ms) I0425 02:58:09.590882 1 gather_workloads_info.go:387] No image sha256:15e1b5f763553e86d0230a751b0783e3b990438fd07cf96eed164bccb72ed564 (104ms) I0425 02:58:09.688860 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (98ms) I0425 02:58:09.788464 1 gather_workloads_info.go:387] No image sha256:e3732e356ae2324565c74cae57d5d016917314fa293ddaa3a68ee9ae030c6f07 (100ms) I0425 02:58:09.887761 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (99ms) I0425 02:58:09.986211 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (98ms) I0425 02:58:10.089631 1 gather_workloads_info.go:387] No image sha256:5ff204630794311b0b37fa7b197b933ad85d76a481bf7cdb3bcbada08f0cdcbf (103ms) I0425 02:58:10.186739 1 gather_workloads_info.go:387] No image sha256:6c6d412ccf0c3d6a14645a79df6c4658c0f8725d7954cb83c3d2f0116499345a (97ms) I0425 02:58:10.289340 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (103ms) I0425 02:58:10.386568 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (97ms) I0425 02:58:10.488652 1 gather_workloads_info.go:398] Found image sha256:23204877f166455ce8f9de9b91a467eeda576f172d5f96124a68aa986c4775d6 (102ms) I0425 02:58:10.588031 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (99ms) I0425 02:58:10.686561 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (98ms) I0425 02:58:10.786163 1 gather_workloads_info.go:387] No image sha256:1eb6a6ee3c66ac15be6bc21bb6015c595c11c15b9dfd21bd9f5ce7f7b990ebaa (100ms) I0425 02:58:10.888668 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (102ms) I0425 02:58:10.994974 1 gather_workloads_info.go:387] No image sha256:3380fecebaeee139c1607fbcd9346cebc92806679568e2244a8905ef0f52aad2 (106ms) I0425 02:58:11.086444 1 gather_workloads_info.go:387] No image sha256:5d6a2b8d2508c7da71192661e7c0dcf1335f48b6336a20189abffbe79eeff962 (91ms) I0425 02:58:11.186028 1 gather_workloads_info.go:387] No image sha256:15928a5b803fa00781b408dbc4c3bfa7c5287b18584ba31f817def8a4430c725 (100ms) I0425 02:58:11.288026 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (102ms) I0425 02:58:11.386915 1 gather_workloads_info.go:387] No image sha256:17aefdc2f4752c23dc8f0947bb01fc34635822eeb18f2d8a5eddf8211b84d450 (99ms) I0425 02:58:11.486393 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (99ms) I0425 02:58:11.585746 1 gather_workloads_info.go:387] No image sha256:bdeb1c455cffccdb6c90db238e9d371de06340b1d6105d47f66d87293f878ac1 (99ms) I0425 02:58:11.689793 1 gather_workloads_info.go:387] No image sha256:52db05084f2a2e7d7bf4040e4865df88ec4074efbbe59a78d861be2c5ab2aa40 (104ms) I0425 02:58:11.786870 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (97ms) I0425 02:58:11.887995 1 gather_workloads_info.go:387] No image sha256:53ff8efdd8f563421ebc1859ab021a3bbc1306f82d4ff5d9c21bc60ec5f57898 (101ms) I0425 02:58:11.987532 1 gather_workloads_info.go:398] Found image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (100ms) I0425 02:58:12.088320 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (101ms) I0425 02:58:12.185145 1 gather_workloads_info.go:387] No image sha256:48f5559439bea030a6423652da448c25fff0c62ba293eb521843c1a8eb89ddb0 (97ms) I0425 02:58:12.285601 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (100ms) I0425 02:58:12.285639 1 tasks_processing.go:74] worker 1 stopped. I0425 02:58:12.285936 1 recorder.go:75] Recording config/workload_info with fingerprint=e57987d497ce20c9aa73fb3d9ebc5ab63b34c271f292f7b8642869a868a7e7fd I0425 02:58:12.285954 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.173944652s to process 1 records I0425 02:58:12.285970 1 periodic.go:263] Periodic gather workloads completed in 8.174s I0425 02:58:12.285978 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0425 02:58:12.285983 1 periodic.go:216] Running conditional gatherer I0425 02:58:12.293380 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0425 02:58:12.455679 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0425 02:58:13.529936 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0425 02:58:13.536790 1 conditional_gatherer.go:392] cluster version is '4.21.9' I0425 02:58:13.536825 1 tasks_processing.go:45] number of workers: 3 I0425 02:58:13.536835 1 tasks_processing.go:69] worker 2 listening for tasks. I0425 02:58:13.536839 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0425 02:58:13.536847 1 tasks_processing.go:69] worker 0 listening for tasks. I0425 02:58:13.536862 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0425 02:58:13.536868 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0425 02:58:13.536868 1 tasks_processing.go:69] worker 1 listening for tasks. I0425 02:58:13.536877 1 tasks_processing.go:74] worker 1 stopped. I0425 02:58:13.536925 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0425 02:58:13.536941 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.088µs to process 1 records I0425 02:58:13.536978 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0425 02:58:13.536987 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.473µs to process 1 records I0425 02:58:13.536992 1 tasks_processing.go:74] worker 0 stopped. I0425 02:58:13.537195 1 tasks_processing.go:74] worker 2 stopped. I0425 02:58:13.537209 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 321.141µs to process 0 records I0425 02:58:13.537225 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0425 02:58:13.537243 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0425 02:58:13.566040 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=fff0e7905fd6a8f85e6b8d6d0face645793cf6bd639e4e0a25c04d376b1f77c6 I0425 02:58:13.566229 1 diskrecorder.go:70] Writing 111 records to /var/lib/insights-operator/insights-2026-04-25-025813.tar.gz I0425 02:58:13.574697 1 diskrecorder.go:51] Wrote 111 records to disk in 8ms I0425 02:58:13.574745 1 periodic.go:285] Gathering cluster info every 2h0m0s I0425 02:58:13.574763 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0425 02:59:20.499073 1 diskrecorder.go:223] Found files to send: insights-2026-04-25-025813.tar.gz I0425 02:59:20.499106 1 insightsuploader.go:150] Checking archives to upload periodically every 15m5.357450351s I0425 02:59:20.499141 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0425 02:59:20.511044 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0425 02:59:20.718919 1 requests.go:87] Successfully reported id=2026-04-25T02:59:20Z x-rh-insights-request-id=0f526c5120a04adda3e5d986ccce9b69, wrote=49307 I0425 02:59:20.719047 1 insightsuploader.go:187] Uploaded report successfully in 219.923091ms I0425 02:59:20.719075 1 controller.go:129] Initializing last reported time to 2026-04-25T02:59:20Z I0425 02:59:20.719096 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0425 02:59:20.719106 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0425 02:59:20.719137 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0425 02:59:20.726025 1 controller.go:554] The operator is healthy I0425 02:59:50.522995 1 controller.go:554] The operator is healthy I0425 03:00:22.522411 1 insightsreport.go:137] Pulling report from smart-proxy I0425 03:00:22.522445 1 insightsreport.go:149] Retrieving report I0425 03:00:22.530541 1 requests.go:110] Retrieving report for cluster: 68fdb819-4c8f-4dc9-b67d-922f2ab53580 I0425 03:00:22.530556 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/68fdb819-4c8f-4dc9-b67d-922f2ab53580/reports I0425 03:00:22.533742 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/68fdb819-4c8f-4dc9-b67d-922f2ab53580/reports I0425 03:00:22.702238 1 insightsreport.go:184] Report retrieved I0425 03:00:22.731132 1 insightsreport.go:239] Report retrieved correctly I0425 03:01:50.524295 1 controller.go:554] The operator is healthy I0425 03:02:50.484160 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0425 03:02:50.494522 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0425 03:02:50.494547 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0425 03:02:50.500467 1 secretconfigobserver.go:119] support secret does not exist I0425 03:03:50.525431 1 controller.go:554] The operator is healthy I0425 03:05:50.520795 1 controller.go:554] The operator is healthy I0425 03:07:50.482199 1 configmapobserver.go:84] configmaps "insights-config" not found I0425 03:07:50.501268 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0425 03:07:50.507243 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0425 03:07:50.507269 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0425 03:07:50.512717 1 secretconfigobserver.go:119] support secret does not exist I0425 03:07:50.518992 1 controller.go:554] The operator is healthy I0425 03:09:50.521400 1 controller.go:554] The operator is healthy I0425 03:11:50.524768 1 controller.go:554] The operator is healthy I0425 03:12:50.512906 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0425 03:12:50.524512 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0425 03:12:50.524537 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0425 03:12:50.530030 1 secretconfigobserver.go:119] support secret does not exist I0425 03:13:50.523858 1 controller.go:554] The operator is healthy I0425 03:14:31.077885 1 insightsuploader.go:146] Nothing to report since 2026-04-25T02:59:20Z I0425 03:15:50.522789 1 controller.go:554] The operator is healthy I0425 03:17:50.484061 1 configmapobserver.go:84] configmaps "insights-config" not found I0425 03:17:50.489625 1 configmapobserver.go:84] configmaps "insights-config" not found I0425 03:17:50.519902 1 controller.go:554] The operator is healthy I0425 03:17:50.530864 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0425 03:17:50.536479 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0425 03:17:50.536502 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0425 03:17:50.541480 1 secretconfigobserver.go:119] support secret does not exist I0425 03:19:50.520933 1 controller.go:554] The operator is healthy I0425 03:21:50.519173 1 controller.go:554] The operator is healthy I0425 03:22:50.541548 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0425 03:22:50.548625 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0425 03:22:50.548672 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0425 03:22:50.554145 1 secretconfigobserver.go:119] support secret does not exist I0425 03:23:50.520286 1 controller.go:554] The operator is healthy