I0408 02:41:58.884075 1 cmd.go:253] Using service-serving-cert provided certificates I0408 02:41:58.884544 1 observer_polling.go:159] Starting file observer I0408 02:41:59.353454 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0408 02:41:59.353646 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0408 02:41:59.354227 1 secure_serving.go:57] Forcing use of http/1.1 only I0408 02:41:59.354237 1 simple_featuregate_reader.go:171] Starting feature-gate-detector W0408 02:41:59.354246 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0408 02:41:59.354251 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0408 02:41:59.354254 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0408 02:41:59.354258 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0408 02:41:59.354260 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0408 02:41:59.354263 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0408 02:41:59.357677 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0408 02:41:59.357716 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"b890c181-a9ea-42d2-8ccd-e85a630a3569", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0408 02:41:59.358482 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0408 02:41:59.358497 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0408 02:41:59.358505 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0408 02:41:59.358503 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0408 02:41:59.358523 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0408 02:41:59.358517 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0408 02:41:59.358887 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0408 02:41:59.359207 1 secure_serving.go:213] Serving securely on [::]:8443 I0408 02:41:59.359231 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0408 02:41:59.362005 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0408 02:41:59.362037 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0408 02:41:59.362145 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0408 02:41:59.366777 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0408 02:41:59.366793 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0408 02:41:59.370723 1 secretconfigobserver.go:119] support secret does not exist I0408 02:41:59.374312 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0408 02:41:59.377631 1 secretconfigobserver.go:119] support secret does not exist I0408 02:41:59.379878 1 recorder.go:161] Pruning old reports every 4h6m55s, max age is 288h0m0s I0408 02:41:59.383888 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0408 02:41:59.383896 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0408 02:41:59.383904 1 insightsreport.go:296] Starting report retriever I0408 02:41:59.383903 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0408 02:41:59.383910 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0408 02:41:59.383940 1 periodic.go:209] Running clusterconfig gatherer I0408 02:41:59.383996 1 tasks_processing.go:45] number of workers: 64 I0408 02:41:59.384021 1 tasks_processing.go:69] worker 2 listening for tasks. I0408 02:41:59.384029 1 tasks_processing.go:69] worker 0 listening for tasks. I0408 02:41:59.384032 1 tasks_processing.go:69] worker 1 listening for tasks. I0408 02:41:59.384040 1 tasks_processing.go:69] worker 25 listening for tasks. I0408 02:41:59.384041 1 tasks_processing.go:69] worker 31 listening for tasks. I0408 02:41:59.384046 1 tasks_processing.go:69] worker 18 listening for tasks. I0408 02:41:59.384044 1 tasks_processing.go:69] worker 17 listening for tasks. I0408 02:41:59.384052 1 tasks_processing.go:69] worker 37 listening for tasks. I0408 02:41:59.384055 1 tasks_processing.go:69] worker 20 listening for tasks. I0408 02:41:59.384061 1 tasks_processing.go:69] worker 3 listening for tasks. I0408 02:41:59.384063 1 tasks_processing.go:71] worker 37 working on pdbs task. I0408 02:41:59.384066 1 tasks_processing.go:69] worker 36 listening for tasks. I0408 02:41:59.384071 1 tasks_processing.go:69] worker 21 listening for tasks. I0408 02:41:59.384073 1 tasks_processing.go:71] worker 36 working on authentication task. I0408 02:41:59.384077 1 tasks_processing.go:69] worker 22 listening for tasks. I0408 02:41:59.384078 1 tasks_processing.go:69] worker 28 listening for tasks. I0408 02:41:59.384077 1 tasks_processing.go:69] worker 34 listening for tasks. I0408 02:41:59.384087 1 tasks_processing.go:69] worker 24 listening for tasks. I0408 02:41:59.384088 1 tasks_processing.go:69] worker 30 listening for tasks. I0408 02:41:59.384049 1 tasks_processing.go:69] worker 26 listening for tasks. I0408 02:41:59.384087 1 tasks_processing.go:69] worker 29 listening for tasks. I0408 02:41:59.384068 1 tasks_processing.go:71] worker 3 working on image task. I0408 02:41:59.384109 1 tasks_processing.go:69] worker 9 listening for tasks. I0408 02:41:59.384110 1 tasks_processing.go:69] worker 8 listening for tasks. I0408 02:41:59.384051 1 tasks_processing.go:69] worker 19 listening for tasks. I0408 02:41:59.384055 1 tasks_processing.go:69] worker 27 listening for tasks. I0408 02:41:59.384062 1 tasks_processing.go:69] worker 35 listening for tasks. I0408 02:41:59.384083 1 tasks_processing.go:69] worker 23 listening for tasks. I0408 02:41:59.384100 1 tasks_processing.go:69] worker 33 listening for tasks. I0408 02:41:59.384100 1 tasks_processing.go:69] worker 32 listening for tasks. I0408 02:41:59.384134 1 tasks_processing.go:69] worker 5 listening for tasks. I0408 02:41:59.384141 1 tasks_processing.go:69] worker 49 listening for tasks. I0408 02:41:59.384141 1 tasks_processing.go:69] worker 15 listening for tasks. I0408 02:41:59.384061 1 tasks_processing.go:71] worker 20 working on tsdb_status task. I0408 02:41:59.384150 1 tasks_processing.go:69] worker 62 listening for tasks. I0408 02:41:59.384156 1 tasks_processing.go:69] worker 14 listening for tasks. I0408 02:41:59.384172 1 tasks_processing.go:69] worker 12 listening for tasks. I0408 02:41:59.384181 1 tasks_processing.go:69] worker 13 listening for tasks. I0408 02:41:59.384186 1 tasks_processing.go:69] worker 58 listening for tasks. I0408 02:41:59.384220 1 tasks_processing.go:69] worker 56 listening for tasks. I0408 02:41:59.384220 1 tasks_processing.go:69] worker 7 listening for tasks. I0408 02:41:59.384229 1 tasks_processing.go:69] worker 57 listening for tasks. I0408 02:41:59.384238 1 tasks_processing.go:69] worker 60 listening for tasks. I0408 02:41:59.384241 1 tasks_processing.go:69] worker 59 listening for tasks. I0408 02:41:59.384247 1 tasks_processing.go:69] worker 52 listening for tasks. I0408 02:41:59.384248 1 tasks_processing.go:69] worker 43 listening for tasks. I0408 02:41:59.384254 1 tasks_processing.go:69] worker 44 listening for tasks. I0408 02:41:59.384076 1 tasks_processing.go:69] worker 4 listening for tasks. I0408 02:41:59.384344 1 tasks_processing.go:69] worker 40 listening for tasks. I0408 02:41:59.384352 1 tasks_processing.go:69] worker 46 listening for tasks. I0408 02:41:59.384363 1 tasks_processing.go:71] worker 46 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0408 02:41:59.384366 1 tasks_processing.go:69] worker 63 listening for tasks. I0408 02:41:59.384372 1 tasks_processing.go:71] worker 22 working on schedulers task. I0408 02:41:59.384368 1 tasks_processing.go:71] worker 21 working on cost_management_metrics_configs task. I0408 02:41:59.384434 1 tasks_processing.go:69] worker 39 listening for tasks. I0408 02:41:59.384446 1 tasks_processing.go:69] worker 38 listening for tasks. I0408 02:41:59.384447 1 tasks_processing.go:69] worker 42 listening for tasks. I0408 02:41:59.384451 1 tasks_processing.go:69] worker 50 listening for tasks. I0408 02:41:59.384457 1 tasks_processing.go:71] worker 50 working on machine_configs task. I0408 02:41:59.384461 1 tasks_processing.go:71] worker 63 working on container_images task. I0408 02:41:59.384555 1 tasks_processing.go:71] worker 28 working on lokistack task. I0408 02:41:59.384563 1 tasks_processing.go:69] worker 51 listening for tasks. I0408 02:41:59.384576 1 tasks_processing.go:69] worker 10 listening for tasks. I0408 02:41:59.384581 1 tasks_processing.go:71] worker 49 working on support_secret task. I0408 02:41:59.384604 1 tasks_processing.go:71] worker 39 working on overlapping_namespace_uids task. I0408 02:41:59.384718 1 tasks_processing.go:71] worker 15 working on nodenetworkconfigurationpolicies task. I0408 02:41:59.384740 1 tasks_processing.go:69] worker 48 listening for tasks. I0408 02:41:59.384759 1 tasks_processing.go:71] worker 38 working on machine_config_pools task. I0408 02:41:59.384817 1 tasks_processing.go:71] worker 2 working on storage_cluster task. I0408 02:41:59.384868 1 tasks_processing.go:71] worker 62 working on qemu_kubevirt_launcher_logs task. I0408 02:41:59.384944 1 tasks_processing.go:69] worker 61 listening for tasks. I0408 02:41:59.384957 1 tasks_processing.go:69] worker 53 listening for tasks. I0408 02:41:59.384964 1 tasks_processing.go:71] worker 53 working on clusterroles task. I0408 02:41:59.384458 1 tasks_processing.go:71] worker 42 working on config_maps task. I0408 02:41:59.385035 1 tasks_processing.go:71] worker 0 working on image_registries task. I0408 02:41:59.385105 1 tasks_processing.go:71] worker 8 working on dvo_metrics task. I0408 02:41:59.385128 1 tasks_processing.go:69] worker 55 listening for tasks. I0408 02:41:59.385136 1 tasks_processing.go:71] worker 55 working on sap_config task. I0408 02:41:59.385150 1 tasks_processing.go:71] worker 57 working on openstack_version task. I0408 02:41:59.385234 1 tasks_processing.go:71] worker 60 working on openstack_controlplanes task. I0408 02:41:59.385258 1 tasks_processing.go:71] worker 51 working on sap_datahubs task. I0408 02:41:59.385270 1 tasks_processing.go:69] worker 47 listening for tasks. I0408 02:41:59.385346 1 tasks_processing.go:71] worker 14 working on operators_pods_and_events task. I0408 02:41:59.385473 1 tasks_processing.go:71] worker 59 working on openshift_logging task. I0408 02:41:59.385495 1 tasks_processing.go:71] worker 1 working on olm_operators task. I0408 02:41:59.385519 1 tasks_processing.go:71] worker 58 working on openstack_dataplanedeployments task. I0408 02:41:59.385546 1 tasks_processing.go:71] worker 34 working on oauths task. I0408 02:41:59.385563 1 tasks_processing.go:71] worker 31 working on ceph_cluster task. I0408 02:41:59.385610 1 tasks_processing.go:69] worker 41 listening for tasks. I0408 02:41:59.385627 1 tasks_processing.go:71] worker 23 working on proxies task. I0408 02:41:59.385639 1 tasks_processing.go:69] worker 6 listening for tasks. I0408 02:41:59.385651 1 tasks_processing.go:71] worker 18 working on version task. I0408 02:41:59.384145 1 tasks_processing.go:69] worker 16 listening for tasks. I0408 02:41:59.385658 1 tasks_processing.go:71] worker 33 working on cluster_apiserver task. I0408 02:41:59.385665 1 tasks_processing.go:71] worker 25 working on openstack_dataplanenodesets task. I0408 02:41:59.385707 1 tasks_processing.go:71] worker 12 working on sap_pods task. I0408 02:41:59.385717 1 tasks_processing.go:71] worker 48 working on storage_classes task. I0408 02:41:59.385722 1 tasks_processing.go:71] worker 30 working on machine_healthchecks task. I0408 02:41:59.385753 1 tasks_processing.go:71] worker 19 working on jaegers task. I0408 02:41:59.385774 1 tasks_processing.go:71] worker 13 working on active_alerts task. I0408 02:41:59.385844 1 tasks_processing.go:71] worker 43 working on node_logs task. I0408 02:41:59.385872 1 tasks_processing.go:71] worker 27 working on silenced_alerts task. I0408 02:41:59.385879 1 tasks_processing.go:71] worker 26 working on install_plans task. I0408 02:41:59.385888 1 tasks_processing.go:71] worker 61 working on infrastructures task. I0408 02:41:59.385946 1 tasks_processing.go:71] worker 56 working on aggregated_monitoring_cr_names task. I0408 02:41:59.385952 1 tasks_processing.go:71] worker 32 working on service_accounts task. I0408 02:41:59.385990 1 tasks_processing.go:71] worker 17 working on machines task. I0408 02:41:59.386050 1 tasks_processing.go:71] worker 35 working on certificate_signing_requests task. I0408 02:41:59.386073 1 tasks_processing.go:71] worker 9 working on validating_webhook_configurations task. I0408 02:41:59.386115 1 tasks_processing.go:71] worker 7 working on openshift_machine_api_events task. I0408 02:41:59.386145 1 tasks_processing.go:71] worker 44 working on ingress task. I0408 02:41:59.386161 1 tasks_processing.go:71] worker 10 working on feature_gates task. I0408 02:41:59.384362 1 tasks_processing.go:71] worker 40 working on machine_autoscalers task. I0408 02:41:59.385718 1 tasks_processing.go:71] worker 24 working on container_runtime_configs task. I0408 02:41:59.386581 1 tasks_processing.go:69] worker 11 listening for tasks. I0408 02:41:59.386644 1 tasks_processing.go:71] worker 29 working on mutating_webhook_configurations task. I0408 02:41:59.386728 1 tasks_processing.go:71] worker 5 working on machine_sets task. I0408 02:41:59.386771 1 tasks_processing.go:71] worker 4 working on image_pruners task. I0408 02:41:59.386749 1 tasks_processing.go:69] worker 54 listening for tasks. I0408 02:41:59.386889 1 tasks_processing.go:71] worker 52 working on nodes task. I0408 02:41:59.386803 1 tasks_processing.go:69] worker 45 listening for tasks. I0408 02:41:59.386985 1 tasks_processing.go:71] worker 45 working on monitoring_persistent_volumes task. I0408 02:41:59.387147 1 tasks_processing.go:71] worker 47 working on crds task. I0408 02:41:59.387228 1 tasks_processing.go:71] worker 16 working on operators task. I0408 02:41:59.387322 1 tasks_processing.go:71] worker 11 working on metrics task. I0408 02:41:59.387338 1 tasks_processing.go:71] worker 41 working on pod_network_connectivity_checks task. I0408 02:41:59.387357 1 tasks_processing.go:71] worker 54 working on nodenetworkstates task. I0408 02:41:59.387469 1 tasks_processing.go:71] worker 6 working on ingress_certificates task. I0408 02:41:59.390206 1 tasks_processing.go:71] worker 21 working on networks task. I0408 02:41:59.390217 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 5.818675ms to process 0 records I0408 02:41:59.390245 1 tasks_processing.go:74] worker 2 stopped. I0408 02:41:59.390256 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 5.413975ms to process 0 records I0408 02:41:59.390446 1 tasks_processing.go:74] worker 15 stopped. I0408 02:41:59.390462 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 5.714165ms to process 0 records I0408 02:41:59.390471 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 5.890065ms to process 0 records I0408 02:41:59.390478 1 tasks_processing.go:74] worker 28 stopped. I0408 02:41:59.390549 1 tasks_processing.go:74] worker 3 stopped. I0408 02:41:59.390696 1 recorder.go:75] Recording config/image with fingerprint=86bffad9d3bba127e130b6ade39b456d044727c28b6e0690a435c3bd6da7c14e I0408 02:41:59.390715 1 gather.go:177] gatherer "clusterconfig" function "image" took 6.439359ms to process 1 records I0408 02:41:59.390739 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0408 02:41:59.390758 1 controller.go:326] The initial operator extension status is healthy I0408 02:41:59.390766 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0408 02:41:59.390772 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0408 02:41:59.390775 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0408 02:41:59.390788 1 controller.go:489] The operator is still being initialized I0408 02:41:59.390794 1 controller.go:512] The operator is healthy I0408 02:41:59.390804 1 tasks_processing.go:74] worker 22 stopped. I0408 02:41:59.390825 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=6552dda65ac03adf56373a6ec9b97b73c9fc1b24e0d47e66567db59bccc04dca I0408 02:41:59.390839 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 6.332815ms to process 1 records I0408 02:41:59.390855 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0408 02:41:59.390880 1 tasks_processing.go:74] worker 36 stopped. I0408 02:41:59.390855 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0408 02:41:59.390966 1 operator.go:288] started I0408 02:41:59.390993 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0408 02:41:59.391126 1 recorder.go:75] Recording config/authentication with fingerprint=ec5ee0d16531bd19741651183c0f6f6cb2c055832470741a09963683771e26c1 I0408 02:41:59.391145 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 6.800749ms to process 1 records E0408 02:41:59.391549 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.144.95:9091: connect: connection refused I0408 02:41:59.391569 1 tasks_processing.go:74] worker 20 stopped. E0408 02:41:59.391575 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.144.95:9091: connect: connection refused I0408 02:41:59.391583 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 7.41187ms to process 0 records E0408 02:41:59.391578 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.144.95:9091: connect: connection refused I0408 02:41:59.391595 1 tasks_processing.go:74] worker 11 stopped. E0408 02:41:59.391602 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.144.95:9091: connect: connection refused I0408 02:41:59.391615 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 4.26207ms to process 0 records I0408 02:41:59.393002 1 tasks_processing.go:74] worker 31 stopped. I0408 02:41:59.393011 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 7.429086ms to process 0 records I0408 02:41:59.393027 1 tasks_processing.go:74] worker 57 stopped. I0408 02:41:59.393036 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 7.86455ms to process 0 records I0408 02:41:59.393083 1 tasks_processing.go:74] worker 38 stopped. I0408 02:41:59.393099 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 8.305205ms to process 0 records I0408 02:41:59.393201 1 tasks_processing.go:74] worker 37 stopped. I0408 02:41:59.393237 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=5a7a845fd96caf112886cd2deb1502eba7f4384a52d3926834a507b625414f2d I0408 02:41:59.393263 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=fe14c8f96ca5df4897f4271c68f5defac6b28a1e8878f920a980ffdbcfc5e90e I0408 02:41:59.393304 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=76b2afb5fb4661e7b6605fe64d327b211cb6ec070ba07378337ae6f90f301db7 I0408 02:41:59.393329 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=48c002ecd065b05fb979529e16a41b28a6b43e51cb0523c94ff50d47922dca5d I0408 02:41:59.393357 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=11bea011c8a91c2536a4fc8fc006bdb03cda125c6e97208964a58b4c632e0531 I0408 02:41:59.393380 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=ebea7bbb2814875ff32130b177b38ac4f893e44c1d018688eda829b1bd1acf08 I0408 02:41:59.393408 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=b127a9d80a54b54652e56b0485c107b4648a8fd0a71e32976dd710f86cdf1723 I0408 02:41:59.393434 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=9bed4a63fff437fef0a9f0bb120f4d0cbed412f24382a03882000a255665a1a0 I0408 02:41:59.393456 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=61abd61e594eba5dd07cd1e44ad3054c15d64f788b6eebfd121dcbafae259920 I0408 02:41:59.393482 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=444ad0b7683de83c7cfbb929cb2cd43cddb6bcaf81e1d15b9399cbc519d422af I0408 02:41:59.393508 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=e88554d14915572dfe645c31458923ef522e95117ab3fae3c0a96b17dd188673 I0408 02:41:59.393520 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 9.007292ms to process 11 records I0408 02:41:59.397449 1 tasks_processing.go:74] worker 51 stopped. I0408 02:41:59.397463 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 12.176429ms to process 0 records I0408 02:41:59.397469 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 12.200455ms to process 0 records I0408 02:41:59.397473 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 11.683229ms to process 0 records I0408 02:41:59.397477 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 11.775325ms to process 0 records I0408 02:41:59.397482 1 tasks_processing.go:74] worker 25 stopped. I0408 02:41:59.397485 1 tasks_processing.go:74] worker 60 stopped. I0408 02:41:59.397489 1 tasks_processing.go:74] worker 19 stopped. I0408 02:41:59.397579 1 tasks_processing.go:74] worker 30 stopped. E0408 02:41:59.397595 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0408 02:41:59.397605 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 11.839562ms to process 0 records I0408 02:41:59.406217 1 tasks_processing.go:74] worker 5 stopped. I0408 02:41:59.406232 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 19.431337ms to process 0 records I0408 02:41:59.406244 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 20.672062ms to process 0 records I0408 02:41:59.406251 1 tasks_processing.go:74] worker 58 stopped. I0408 02:41:59.410255 1 tasks_processing.go:74] worker 55 stopped. I0408 02:41:59.410268 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 25.111606ms to process 0 records I0408 02:41:59.410298 1 tasks_processing.go:74] worker 59 stopped. I0408 02:41:59.410307 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 24.810886ms to process 0 records I0408 02:41:59.410336 1 tasks_processing.go:74] worker 17 stopped. E0408 02:41:59.410351 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0408 02:41:59.410358 1 gather.go:177] gatherer "clusterconfig" function "machines" took 24.330517ms to process 0 records I0408 02:41:59.410403 1 gather_logs.go:145] no pods in namespace were found I0408 02:41:59.410411 1 tasks_processing.go:74] worker 62 stopped. I0408 02:41:59.410417 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 25.534629ms to process 0 records I0408 02:41:59.410554 1 tasks_processing.go:74] worker 61 stopped. I0408 02:41:59.411065 1 recorder.go:75] Recording config/infrastructure with fingerprint=de94332102ddf728998634703e232dae245f06bbf4e3ecfc34be6703aece703b I0408 02:41:59.411094 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 24.655791ms to process 1 records I0408 02:41:59.411112 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 23.301959ms to process 0 records I0408 02:41:59.411122 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 23.762913ms to process 0 records I0408 02:41:59.411168 1 tasks_processing.go:74] worker 54 stopped. I0408 02:41:59.411182 1 tasks_processing.go:74] worker 45 stopped. I0408 02:41:59.411197 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=08446acbac3b7904e5a5a627bc10c5d0c308d5193b8e91dd7d30d9eb954f96a6 I0408 02:41:59.411216 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=f89d4b8679dd047543a9de7e453d6dc662356af595e627cd941d9413a655ba1b I0408 02:41:59.411227 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 25.041856ms to process 2 records I0408 02:41:59.411234 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 24.636527ms to process 0 records I0408 02:41:59.411242 1 tasks_processing.go:74] worker 48 stopped. I0408 02:41:59.411242 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 24.373982ms to process 0 records I0408 02:41:59.411250 1 tasks_processing.go:74] worker 24 stopped. I0408 02:41:59.411250 1 tasks_processing.go:74] worker 40 stopped. I0408 02:41:59.411291 1 tasks_processing.go:74] worker 4 stopped. I0408 02:41:59.411448 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=ae714ebff57e7b8ce02f7951cd9c9de393bba81c00fa2f50918e4f54377a37cf I0408 02:41:59.411461 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 24.264224ms to process 1 records I0408 02:41:59.411467 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 25.605251ms to process 0 records I0408 02:41:59.411472 1 tasks_processing.go:74] worker 43 stopped. I0408 02:41:59.412079 1 tasks_processing.go:74] worker 39 stopped. I0408 02:41:59.412100 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0408 02:41:59.412109 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 27.457548ms to process 1 records I0408 02:41:59.412215 1 tasks_processing.go:74] worker 12 stopped. I0408 02:41:59.412227 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 26.497793ms to process 0 records E0408 02:41:59.412234 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0408 02:41:59.412238 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 24.876012ms to process 0 records I0408 02:41:59.412245 1 tasks_processing.go:74] worker 41 stopped. I0408 02:41:59.412268 1 tasks_processing.go:74] worker 7 stopped. I0408 02:41:59.412295 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 26.13974ms to process 0 records I0408 02:41:59.412960 1 tasks_processing.go:74] worker 49 stopped. E0408 02:41:59.412974 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0408 02:41:59.412984 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 28.342893ms to process 0 records I0408 02:41:59.413016 1 tasks_processing.go:74] worker 23 stopped. I0408 02:41:59.413084 1 recorder.go:75] Recording config/proxy with fingerprint=cc1171d896d38a5b2d2b7da76b7997c844b7e8a331b37eec0ed1c2ec7fd5d4a0 I0408 02:41:59.413097 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 27.380732ms to process 1 records I0408 02:41:59.413103 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 27.568902ms to process 0 records I0408 02:41:59.413108 1 tasks_processing.go:74] worker 1 stopped. I0408 02:41:59.413443 1 tasks_processing.go:74] worker 44 stopped. I0408 02:41:59.413781 1 recorder.go:75] Recording config/ingress with fingerprint=e8c5e4b0a2cafd5984f4a0d48ceaa03ea743ae0315aadab1efb726cf5583a4f6 I0408 02:41:59.413794 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 27.287273ms to process 1 records I0408 02:41:59.413945 1 tasks_processing.go:74] worker 34 stopped. I0408 02:41:59.414244 1 recorder.go:75] Recording config/oauth with fingerprint=0707b9586fca597002a79bb90dda608b392311fb730644e21d155d213d04e37f I0408 02:41:59.414323 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 28.151277ms to process 1 records I0408 02:41:59.414497 1 tasks_processing.go:74] worker 0 stopped. I0408 02:41:59.415153 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=19903b53f2b72f3fa86ffb46f4e9060667333e495cef9dcaf904de31dd31b767 I0408 02:41:59.415169 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 29.075944ms to process 1 records I0408 02:41:59.415343 1 recorder.go:75] Recording config/apiserver with fingerprint=7e4f122392ef34851629e5e630dac9118c1a04c0d1bdd3c94a8bd63c3b01482c I0408 02:41:59.415358 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 28.469466ms to process 1 records I0408 02:41:59.416019 1 tasks_processing.go:74] worker 33 stopped. I0408 02:41:59.416025 1 tasks_processing.go:74] worker 29 stopped. W0408 02:41:59.416063 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0408 02:41:59.416239 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=9794162622d5787f4563d6ab849ffec73cd67997d0f39c0a3928ba6b001163b2 I0408 02:41:59.416301 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=a4c12bf019279466dbb12d472117637afd584163ff8b02ce363ffc126bf5c29a I0408 02:41:59.416329 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=c2b9de5050d86b566559867dbfdd6288a179e7cda5b35b42d15fd6e5793a596c I0408 02:41:59.416338 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 29.314171ms to process 3 records I0408 02:41:59.416413 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=97689b20c26c5128f3bf4f9c62542738d4d65a510537c557dcdf18838c6b0dab I0408 02:41:59.416422 1 tasks_processing.go:74] worker 9 stopped. I0408 02:41:59.416481 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=4a9ac40b96787d7c16d7917724bdb9d5d0c724b85e4f109471e5c1c0f4b318a2 I0408 02:41:59.416616 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=978707d8321739001d438f687872f49bf92bfc7ea6366c4c73d11697e3905292 I0408 02:41:59.416658 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=1a5f0fec52dea3c843f0d512f24e8dd9d9c865e6a6bcf939ef58421ccdecd9b3 I0408 02:41:59.416697 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=b5ba87032fad6a1e1aa3ae81d0b34a37c8a6ea835957880d18f77131bf62042e I0408 02:41:59.416736 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=8fb7f168ddc5f3d3918d1ba2b3664fe7f5ae7b4971ec438e9e11d5c4be1e454d I0408 02:41:59.416775 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=9d563e879bf5ff0b84afe8251d15521b2ec1f7c4866128146a7d015a72351def I0408 02:41:59.416837 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=6a490e479aa2c8347ed7ebfbd6a72c03fa73c675119267ebbe46dab76d3f9956 I0408 02:41:59.416876 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=fe87d14b0eb61c49f6072216077d29be4f64bc38e7831e67872db5c69fa30aad I0408 02:41:59.416936 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=31507dceebe624331f97e6b53fce81142dd183028c56d398db507e0f4d36d661 I0408 02:41:59.416978 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=9e43b0a4050cc55ffab53dec738b22afd142d7f6ed4d012a0d2e8375bc25a11a I0408 02:41:59.417005 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0408 02:41:59.417018 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0408 02:41:59.417018 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=a75367ef0dc045385ec806a89a4c8b5eee3029f47d16940be890590deac5ab97 I0408 02:41:59.417021 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0408 02:41:59.417026 1 controller.go:212] Source scaController *sca.Controller is not ready I0408 02:41:59.417040 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0408 02:41:59.417056 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=9cf058323d4292894daa99c26e130c9fd61a25e2978f2fbcc12ac27130b439b8 I0408 02:41:59.417069 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 30.215075ms to process 13 records I0408 02:41:59.417058 1 controller.go:489] The operator is still being initialized I0408 02:41:59.417131 1 controller.go:512] The operator is healthy I0408 02:41:59.417206 1 tasks_processing.go:74] worker 21 stopped. I0408 02:41:59.417232 1 recorder.go:75] Recording config/network with fingerprint=1fbfbb4cd4e9ca7cee200dba340f2c17629bd47babe924abd4ee6b50d2e8cc2f I0408 02:41:59.417243 1 gather.go:177] gatherer "clusterconfig" function "networks" took 26.203377ms to process 1 records I0408 02:41:59.417307 1 tasks_processing.go:74] worker 52 stopped. I0408 02:41:59.417659 1 recorder.go:75] Recording config/node/ip-10-0-0-83.ec2.internal with fingerprint=7749f214810a0cacb3ee6c3ea4f1bf3c043c66a8a48b8155bd66a6c4a4a6a2a5 I0408 02:41:59.417777 1 recorder.go:75] Recording config/node/ip-10-0-1-13.ec2.internal with fingerprint=cf6961dfd28037a5805b2a926bf3dade1ca9995a45927bb5386957f8fd862196 I0408 02:41:59.417878 1 recorder.go:75] Recording config/node/ip-10-0-2-74.ec2.internal with fingerprint=0ec8afaa186c1977f679f19b47e644fd1fb42becd68a6a0b0801dc6ee7d5917e I0408 02:41:59.417889 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 29.547443ms to process 3 records I0408 02:41:59.417971 1 tasks_processing.go:74] worker 10 stopped. I0408 02:41:59.418050 1 recorder.go:75] Recording config/featuregate with fingerprint=55eaa5867f4a7be1eb4553ba0135d7ae32cfaf509f0e97369f87a76df100947d I0408 02:41:59.418085 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 30.815773ms to process 1 records I0408 02:41:59.420670 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 34.30388ms to process 0 records I0408 02:41:59.420691 1 tasks_processing.go:74] worker 35 stopped. I0408 02:41:59.422490 1 tasks_processing.go:74] worker 14 stopped. I0408 02:41:59.422512 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 37.120153ms to process 0 records I0408 02:41:59.427262 1 tasks_processing.go:74] worker 18 stopped. I0408 02:41:59.427608 1 recorder.go:75] Recording config/version with fingerprint=24e9843360d9ff1176fad8924fc43cb17dc564e108cdbb31d5302bb3f8c2e287 I0408 02:41:59.427627 1 recorder.go:75] Recording config/id with fingerprint=8d15f0d3e2b2b6d429b15ebe636274307433ceb8d6b0e089c3b2770f01164dcc I0408 02:41:59.427635 1 gather.go:177] gatherer "clusterconfig" function "version" took 41.598395ms to process 2 records I0408 02:41:59.427707 1 tasks_processing.go:74] worker 53 stopped. I0408 02:41:59.427843 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=910717130598cdea71561554068276a6aed8e33b861efaf2023c232a8927d666 I0408 02:41:59.427992 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=d0dec2387276379a158ea47ec7889697c921e7d211c97a54540b12f4800fe5d3 I0408 02:41:59.428004 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 42.518973ms to process 2 records I0408 02:41:59.428210 1 tasks_processing.go:74] worker 47 stopped. I0408 02:41:59.428854 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=9c4396a340cd883f5a67749dfae54fa87e29fd0c4b4d1f7eb779e0db8821ae7f I0408 02:41:59.429084 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=7f5a618ec57ecf38053d4f0a1ee8a8cc60d8399539c138df9d0068c966181803 I0408 02:41:59.429094 1 gather.go:177] gatherer "clusterconfig" function "crds" took 41.040611ms to process 2 records I0408 02:41:59.432678 1 tasks_processing.go:74] worker 50 stopped. I0408 02:41:59.432726 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0408 02:41:59.432743 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 48.209778ms to process 1 records I0408 02:41:59.433885 1 tasks_processing.go:74] worker 56 stopped. I0408 02:41:59.433962 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 47.922647ms to process 0 records I0408 02:41:59.436994 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0408 02:41:59.440431 1 tasks_processing.go:74] worker 63 stopped. I0408 02:41:59.442768 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-c2qgd with fingerprint=4a0bbef7ed741f8124c66925b05557e176530254aa9ec01fc0cd23222538b5fb I0408 02:41:59.442851 1 recorder.go:75] Recording config/running_containers with fingerprint=03bed6f1668b6cf2c9b1a8b68b733944580349d59e72b4a4d55a594b5dad9cea I0408 02:41:59.442862 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 55.959521ms to process 2 records E0408 02:41:59.442874 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0408 02:41:59.442879 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0408 02:41:59.442882 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0408 02:41:59.442890 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0408 02:41:59.442890 1 tasks_processing.go:74] worker 42 stopped. I0408 02:41:59.442913 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0408 02:41:59.442920 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0408 02:41:59.442938 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0408 02:41:59.442948 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0408 02:41:59.442985 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0408 02:41:59.442993 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0408 02:41:59.442998 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 57.427344ms to process 7 records I0408 02:41:59.445145 1 tasks_processing.go:74] worker 6 stopped. E0408 02:41:59.445156 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0408 02:41:59.445162 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2phms2qjfbs2526um21ukv18g3hh208o-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2phms2qjfbs2526um21ukv18g3hh208o-primary-cert-bundle-secret" not found I0408 02:41:59.445203 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=0b7ae64532ec2e41251dd6a7e4967239872c9dd4ecb0699fd06d9f364d3ce294 I0408 02:41:59.445214 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 57.657298ms to process 1 records I0408 02:41:59.445629 1 tasks_processing.go:74] worker 46 stopped. I0408 02:41:59.445643 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 61.255623ms to process 0 records I0408 02:41:59.458890 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0408 02:41:59.458911 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0408 02:41:59.458922 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0408 02:41:59.462293 1 base_controller.go:82] Caches are synced for ConfigController I0408 02:41:59.462305 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0408 02:41:59.466301 1 configmapobserver.go:84] configmaps "insights-config" not found I0408 02:41:59.491765 1 base_controller.go:82] Caches are synced for LoggingSyncer I0408 02:41:59.491775 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0408 02:41:59.501758 1 cluster_transfer.go:102] no available accepted cluster transfer I0408 02:41:59.501771 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0408 02:41:59.612677 1 sca.go:266] etc-pki-entitlement secret successfully created I0408 02:41:59.612693 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated W0408 02:42:00.414705 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0408 02:42:00.417810 1 tasks_processing.go:74] worker 27 stopped. I0408 02:42:00.417830 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0408 02:42:00.417841 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.031921449s to process 1 records I0408 02:42:00.418090 1 tasks_processing.go:74] worker 13 stopped. I0408 02:42:00.418126 1 recorder.go:75] Recording config/alerts with fingerprint=ea3896c4c9dd49a05daeddccb1d8d709a6f5fc88909863bd80a60da47d4c2200 I0408 02:42:00.418138 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.032299198s to process 1 records W0408 02:42:01.415518 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0408 02:42:01.843365 1 tasks_processing.go:74] worker 16 stopped. I0408 02:42:01.843476 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=91878cd8f4261095a9b55d320f066f84cdd8813dce79b79a7a4d12ec12fc42ae I0408 02:42:01.843518 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0408 02:42:01.843541 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=b1e383dc8de42a2bfeaa2897fd5a28741af814ecd8791ae6273647ecfdd24e59 I0408 02:42:01.843551 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0408 02:42:01.843571 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=1885d889b8d37cf202172d143485b7b01303ba60419eb75de858811fa2198ca8 I0408 02:42:01.843586 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0408 02:42:01.843607 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=3dd87fbb09e31753ea5b5354f20ef23ba14a106e937d24150dc62e53f92fa8b5 I0408 02:42:01.843635 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=62d4d5a9458828d1a71fd62242ae4a092d3063eeab24a251bac24da455cf1a7d I0408 02:42:01.843663 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=34c20d7a829ffd984a4b6098fe488a27517f48583284675750cca01467b4aaac I0408 02:42:01.843693 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=6c1103fcdac32ae557993489b4f9c3cc411c7b9cfcb28b541c5a37b5cf542cb1 I0408 02:42:01.843947 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=f3b1832523b270fb5e0b1c43a8ceae042b7aa7442e47406838d1fb1de96609b6 I0408 02:42:01.843967 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=05618a3454f2fb4f8018be0fe2b21f9dc4b95700f22a3a7fc2036c74ea2d3a07 I0408 02:42:01.843976 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0408 02:42:01.843992 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=627467b4655f6d78b60afa4288f5379a32b09e26878be2456dd706d884e08b7f I0408 02:42:01.844002 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0408 02:42:01.844017 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=0e358b38fac0ff066d7d0a3312ed74a11d8a1b797fdde729515da7c48a7a270c I0408 02:42:01.844026 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0408 02:42:01.844045 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=c472af48f48365bd96c00ae92a6c00a51211d39dbf0cad138a0c30a245a83678 I0408 02:42:01.844055 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0408 02:42:01.844074 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=e363f37c71b291de2339fe3b10b2a606b0b6c9b6cbc833e028e40272d8f2d889 I0408 02:42:01.844178 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=cb22bab6754a25128a754c298853c1777a0d7552eada262068651c1304be2088 I0408 02:42:01.844187 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0408 02:42:01.844194 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0408 02:42:01.844214 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0408 02:42:01.844234 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=8296c89699dd8a8350208887624e6f3ed70fae816d1a0086bf9bc52ee94a59e5 I0408 02:42:01.844258 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=746ec15b296e35451eec39e79fca8aa286478647fc84d58eeebfbf2a3f4ca029 I0408 02:42:01.844267 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0408 02:42:01.844298 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=c199acd81296e2b914f808108eda03c1de734cabfce73b88edb640f509024e20 I0408 02:42:01.844309 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0408 02:42:01.844331 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=e55c14d96208481e7e40839c8108a061338b248f4ed63f5a9049f0d760176516 I0408 02:42:01.844346 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0408 02:42:01.844359 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=2ec633ad79ba25dd7c2fd56de962e400878e3eda40ac3dafb7b47215d8521a7f I0408 02:42:01.844373 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=e53c6c1a521af4693412a2d6c6260b76e0d3af19342b4ccbab5addf93df2bf22 I0408 02:42:01.844387 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=9dc7e6dfd3ecd0963d44bc7dd1d1daa74a84480cb369c6cb893a6fbc28983ed1 I0408 02:42:01.844409 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=38392e933f5f0d64f3e497f009e52a6691b559b85c48dc7d5c7080c7801512c6 I0408 02:42:01.844417 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0408 02:42:01.844440 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=5f819a88a34f92a4868e8e57c1280caf3cbee1cff8c02759387afb5afc8e9cc4 I0408 02:42:01.844456 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0408 02:42:01.844464 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0408 02:42:01.844471 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.456112131s to process 39 records W0408 02:42:02.414747 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0408 02:42:03.414786 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0408 02:42:04.412970 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0408 02:42:04.413005 1 tasks_processing.go:74] worker 8 stopped. E0408 02:42:04.413017 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0408 02:42:04.413029 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0408 02:42:04.413038 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.027877787s to process 1 records I0408 02:42:12.223177 1 tasks_processing.go:74] worker 26 stopped. I0408 02:42:12.223217 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0408 02:42:12.223229 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.837278921s to process 1 records I0408 02:42:12.991882 1 tasks_processing.go:74] worker 32 stopped. I0408 02:42:12.992148 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=230dfa4a1c30b0cc2ffe8f912592f10befbcd8eab365a68f86f7a7783043a01b I0408 02:42:12.992164 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.605914257s to process 1 records E0408 02:42:12.992224 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.608s with: function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0408 02:42:12.993309 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "tsdb_status" failed with an error, function "metrics" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0408 02:42:12.993326 1 periodic.go:209] Running workloads gatherer I0408 02:42:12.993343 1 tasks_processing.go:45] number of workers: 2 I0408 02:42:12.993354 1 tasks_processing.go:69] worker 1 listening for tasks. I0408 02:42:12.993359 1 tasks_processing.go:71] worker 1 working on workload_info task. I0408 02:42:12.993368 1 tasks_processing.go:69] worker 0 listening for tasks. I0408 02:42:12.993394 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0408 02:42:12.998242 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-83.ec2.internal... I0408 02:42:12.998256 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-13.ec2.internal... I0408 02:42:12.998269 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-74.ec2.internal... I0408 02:42:13.017484 1 tasks_processing.go:74] worker 0 stopped. I0408 02:42:13.017500 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 24.076415ms to process 0 records I0408 02:42:18.117703 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 5s I0408 02:42:18.139979 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0408 02:42:18.148650 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (10ms) I0408 02:42:18.157523 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (9ms) I0408 02:42:18.165875 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (8ms) I0408 02:42:18.174304 1 gather_workloads_info.go:398] Found image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (8ms) I0408 02:42:18.182271 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (8ms) I0408 02:42:18.189608 1 gather_workloads_info.go:387] No image sha256:1e65f9f5490fbc930d2dce96f03ddcb28990b839d7dc011ef5de5bec6c5095bf (7ms) I0408 02:42:18.197067 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (7ms) I0408 02:42:18.204356 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (7ms) I0408 02:42:18.211669 1 gather_workloads_info.go:387] No image sha256:1477f3287cc75352305274f248f781c1b7c2a8696f4f7bc450c45a4a5f5b10ea (7ms) I0408 02:42:18.219039 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (7ms) I0408 02:42:18.248698 1 gather_workloads_info.go:387] No image sha256:90bdf9e678fe18e35401e34e1cb0912d822e9cb32b9e41cc545870540e9a87f2 (30ms) I0408 02:42:18.348222 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (100ms) I0408 02:42:18.449384 1 gather_workloads_info.go:387] No image sha256:0af73ad7a47c9ff53269ae2bc5163bf9e389085220c34e4bd42c92c4e7a85f6d (101ms) I0408 02:42:18.548839 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (99ms) I0408 02:42:18.648555 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (100ms) I0408 02:42:18.749068 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0408 02:42:18.852697 1 gather_workloads_info.go:387] No image sha256:7bd3e88d6d3f878a89203df590b684c5a77bf08df4717451d3197ab74683a80e (104ms) I0408 02:42:18.949128 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (96ms) I0408 02:42:19.049693 1 gather_workloads_info.go:387] No image sha256:513b1166e376feca472fdd5174f6abf0abf02925d6a21b7b4d89a2ff2fc67bed (101ms) I0408 02:42:19.148772 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (99ms) I0408 02:42:19.248672 1 gather_workloads_info.go:387] No image sha256:6e5df2aab9ce759e8f957341a11c087c233307e3c8f8760d21270efc19ccd3ce (100ms) I0408 02:42:19.347915 1 gather_workloads_info.go:398] Found image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (99ms) I0408 02:42:19.448921 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (101ms) I0408 02:42:19.548539 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0408 02:42:19.649177 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (101ms) I0408 02:42:19.749437 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (100ms) I0408 02:42:19.849165 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0408 02:42:19.947920 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (99ms) I0408 02:42:20.047964 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0408 02:42:20.148482 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0408 02:42:20.249015 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (101ms) I0408 02:42:20.348933 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (100ms) I0408 02:42:20.448936 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0408 02:42:20.548523 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (100ms) I0408 02:42:20.648835 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (100ms) I0408 02:42:20.755103 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (106ms) I0408 02:42:20.848479 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (93ms) I0408 02:42:20.948175 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (100ms) I0408 02:42:21.048030 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (100ms) I0408 02:42:21.148825 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (101ms) I0408 02:42:21.248473 1 gather_workloads_info.go:387] No image sha256:0092c6ff139f8698c91349d507eb3182bc5d6c4ae040f672e347ff504accdea2 (100ms) I0408 02:42:21.349213 1 gather_workloads_info.go:387] No image sha256:74713675d580a47a56b628469c4c05c41e114e0b935960d718ff0cf7e72f613d (101ms) I0408 02:42:21.448468 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (99ms) I0408 02:42:21.549237 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (101ms) I0408 02:42:21.649620 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0408 02:42:21.749158 1 gather_workloads_info.go:387] No image sha256:b99613494e9d6a4a8207323872d63cc02b830fc80a8bf972e62c64381aa4cd17 (100ms) I0408 02:42:21.848416 1 gather_workloads_info.go:387] No image sha256:101993f6e65d6fda8b90c583b718b720809182961f69bc777295f47e6df4d7ce (99ms) I0408 02:42:21.848449 1 tasks_processing.go:74] worker 1 stopped. I0408 02:42:21.848726 1 recorder.go:75] Recording config/workload_info with fingerprint=77b3583afeea566ba3af8bce2688301eba4ae0ba1e1f520b233b55d5947caab0 I0408 02:42:21.848742 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.855082989s to process 1 records I0408 02:42:21.848758 1 periodic.go:256] Periodic gather workloads completed in 8.855s I0408 02:42:21.848767 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0408 02:42:21.848772 1 periodic.go:209] Running conditional gatherer I0408 02:42:21.854344 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0408 02:42:22.085985 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0408 02:42:23.152101 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0408 02:42:23.157824 1 conditional_gatherer.go:392] cluster version is '4.20.8' I0408 02:42:23.157857 1 tasks_processing.go:45] number of workers: 3 I0408 02:42:23.157865 1 tasks_processing.go:69] worker 2 listening for tasks. I0408 02:42:23.157871 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0408 02:42:23.157879 1 tasks_processing.go:69] worker 0 listening for tasks. I0408 02:42:23.157893 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0408 02:42:23.157899 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0408 02:42:23.157903 1 tasks_processing.go:69] worker 1 listening for tasks. I0408 02:42:23.157915 1 tasks_processing.go:74] worker 1 stopped. I0408 02:42:23.157964 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0408 02:42:23.157980 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.29µs to process 1 records I0408 02:42:23.158013 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0408 02:42:23.158021 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.36µs to process 1 records I0408 02:42:23.158027 1 tasks_processing.go:74] worker 0 stopped. I0408 02:42:23.158173 1 tasks_processing.go:74] worker 2 stopped. I0408 02:42:23.158191 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 267.953µs to process 0 records I0408 02:42:23.158216 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0408 02:42:23.158245 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0408 02:42:23.181493 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=f0bcb3bd7eef6c9de3f304a74880a81d6f284585d0042d1ca582367b7853d9ba I0408 02:42:23.181634 1 diskrecorder.go:70] Writing 110 records to /var/lib/insights-operator/insights-2026-04-08-024223.tar.gz I0408 02:42:23.188471 1 diskrecorder.go:51] Wrote 110 records to disk in 6ms I0408 02:42:23.188503 1 periodic.go:278] Gathering cluster info every 2h0m0s I0408 02:42:23.188522 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0408 02:43:29.384040 1 diskrecorder.go:223] Found files to send: insights-2026-04-08-024223.tar.gz I0408 02:43:29.384072 1 insightsuploader.go:150] Checking archives to upload periodically every 15m34.811833671s I0408 02:43:29.384081 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0408 02:43:29.394531 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0408 02:43:29.629213 1 requests.go:87] Successfully reported id=2026-04-08T02:43:29Z x-rh-insights-request-id=8bc7c4598a2c43b395fc2dad0056889e, wrote=47999 I0408 02:43:29.629309 1 insightsuploader.go:187] Uploaded report successfully in 245.217956ms I0408 02:43:29.629333 1 controller.go:128] Initializing last reported time to 2026-04-08T02:43:29Z I0408 02:43:29.629402 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0408 02:43:29.629416 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0408 02:43:29.629425 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0408 02:43:29.634552 1 controller.go:512] The operator is healthy I0408 02:43:59.398952 1 controller.go:512] The operator is healthy I0408 02:44:31.627385 1 insightsreport.go:137] Pulling report from smart-proxy I0408 02:44:31.627413 1 insightsreport.go:149] Retrieving report I0408 02:44:31.633685 1 requests.go:110] Retrieving report for cluster: cfa8a147-7179-471c-9154-4ed45c14b79f I0408 02:44:31.633699 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/cfa8a147-7179-471c-9154-4ed45c14b79f/reports I0408 02:44:31.636524 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/cfa8a147-7179-471c-9154-4ed45c14b79f/reports I0408 02:44:31.822033 1 insightsreport.go:184] Report retrieved I0408 02:44:31.846557 1 insightsreport.go:239] Report retrieved correctly I0408 02:45:59.399520 1 controller.go:512] The operator is healthy I0408 02:46:59.378460 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0408 02:46:59.390604 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0408 02:46:59.390629 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0408 02:46:59.394911 1 secretconfigobserver.go:119] support secret does not exist I0408 02:47:59.397343 1 controller.go:512] The operator is healthy I0408 02:49:59.397101 1 controller.go:512] The operator is healthy I0408 02:51:59.376697 1 configmapobserver.go:84] configmaps "insights-config" not found I0408 02:51:59.395977 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0408 02:51:59.396898 1 controller.go:512] The operator is healthy I0408 02:51:59.399946 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0408 02:51:59.399961 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0408 02:51:59.404172 1 secretconfigobserver.go:119] support secret does not exist I0408 02:53:59.397076 1 controller.go:512] The operator is healthy I0408 02:55:59.397309 1 controller.go:512] The operator is healthy I0408 02:56:59.404235 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0408 02:56:59.412382 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0408 02:56:59.412402 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0408 02:56:59.416037 1 secretconfigobserver.go:119] support secret does not exist I0408 02:57:59.400019 1 controller.go:512] The operator is healthy I0408 02:59:09.442919 1 insightsuploader.go:146] Nothing to report since 2026-04-08T02:43:29Z I0408 02:59:59.397422 1 controller.go:512] The operator is healthy I0408 03:01:59.377226 1 configmapobserver.go:84] configmaps "insights-config" not found I0408 03:01:59.396894 1 controller.go:512] The operator is healthy I0408 03:01:59.416897 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0408 03:01:59.421413 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0408 03:01:59.421430 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0408 03:01:59.425143 1 secretconfigobserver.go:119] support secret does not exist I0408 03:03:59.396728 1 controller.go:512] The operator is healthy I0408 03:05:59.397106 1 controller.go:512] The operator is healthy