I0320 11:06:35.097756 1 cmd.go:253] Using service-serving-cert provided certificates I0320 11:06:35.098225 1 observer_polling.go:159] Starting file observer I0320 11:06:35.557354 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0320 11:06:35.557550 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0320 11:06:35.558120 1 secure_serving.go:57] Forcing use of http/1.1 only W0320 11:06:35.558141 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0320 11:06:35.558149 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. I0320 11:06:35.558151 1 simple_featuregate_reader.go:171] Starting feature-gate-detector W0320 11:06:35.558155 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0320 11:06:35.558178 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0320 11:06:35.558187 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0320 11:06:35.558194 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0320 11:06:35.563464 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0320 11:06:35.563475 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"7989620d-3db1-40f9-b174-6273572ab932", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0320 11:06:35.564012 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0320 11:06:35.564056 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0320 11:06:35.564055 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0320 11:06:35.564075 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0320 11:06:35.564071 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0320 11:06:35.564091 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0320 11:06:35.564375 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0320 11:06:35.564664 1 secure_serving.go:213] Serving securely on [::]:8443 I0320 11:06:35.564692 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0320 11:06:35.572110 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0320 11:06:35.572132 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0320 11:06:35.572248 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0320 11:06:35.576698 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 11:06:35.576714 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0320 11:06:35.581974 1 secretconfigobserver.go:119] support secret does not exist I0320 11:06:35.586432 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 11:06:35.590756 1 secretconfigobserver.go:119] support secret does not exist I0320 11:06:35.595176 1 recorder.go:176] Pruning old reports every 7h3m13s, max age is 288h0m0s I0320 11:06:35.600271 1 periodic.go:216] Running clusterconfig gatherer I0320 11:06:35.600280 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0320 11:06:35.600297 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0320 11:06:35.600317 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0320 11:06:35.600320 1 tasks_processing.go:45] number of workers: 32 I0320 11:06:35.600321 1 insightsreport.go:296] Starting report retriever I0320 11:06:35.600326 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0320 11:06:35.600338 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 11:06:35.600344 1 tasks_processing.go:69] worker 31 listening for tasks. I0320 11:06:35.600346 1 tasks_processing.go:69] worker 12 listening for tasks. I0320 11:06:35.600348 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 11:06:35.600358 1 tasks_processing.go:71] worker 12 working on overlapping_namespace_uids task. I0320 11:06:35.600359 1 tasks_processing.go:69] worker 21 listening for tasks. I0320 11:06:35.600361 1 tasks_processing.go:69] worker 20 listening for tasks. I0320 11:06:35.600353 1 tasks_processing.go:69] worker 16 listening for tasks. I0320 11:06:35.600369 1 tasks_processing.go:69] worker 15 listening for tasks. I0320 11:06:35.600371 1 tasks_processing.go:69] worker 14 listening for tasks. I0320 11:06:35.600361 1 tasks_processing.go:69] worker 13 listening for tasks. I0320 11:06:35.600380 1 tasks_processing.go:69] worker 18 listening for tasks. I0320 11:06:35.600371 1 tasks_processing.go:69] worker 22 listening for tasks. I0320 11:06:35.600385 1 tasks_processing.go:69] worker 19 listening for tasks. I0320 11:06:35.600386 1 tasks_processing.go:69] worker 4 listening for tasks. I0320 11:06:35.600391 1 tasks_processing.go:69] worker 29 listening for tasks. I0320 11:06:35.600395 1 tasks_processing.go:69] worker 2 listening for tasks. I0320 11:06:35.600397 1 tasks_processing.go:69] worker 26 listening for tasks. I0320 11:06:35.600394 1 tasks_processing.go:69] worker 25 listening for tasks. I0320 11:06:35.600400 1 tasks_processing.go:69] worker 3 listening for tasks. I0320 11:06:35.600403 1 tasks_processing.go:69] worker 30 listening for tasks. I0320 11:06:35.600403 1 tasks_processing.go:69] worker 27 listening for tasks. I0320 11:06:35.600406 1 tasks_processing.go:69] worker 8 listening for tasks. I0320 11:06:35.600351 1 tasks_processing.go:71] worker 31 working on nodenetworkstates task. I0320 11:06:35.600413 1 tasks_processing.go:69] worker 11 listening for tasks. I0320 11:06:35.600416 1 tasks_processing.go:69] worker 6 listening for tasks. I0320 11:06:35.600416 1 tasks_processing.go:69] worker 9 listening for tasks. I0320 11:06:35.600448 1 tasks_processing.go:69] worker 7 listening for tasks. I0320 11:06:35.600481 1 tasks_processing.go:71] worker 7 working on aggregated_monitoring_cr_names task. I0320 11:06:35.600489 1 tasks_processing.go:71] worker 30 working on machine_autoscalers task. I0320 11:06:35.600494 1 tasks_processing.go:71] worker 1 working on image task. I0320 11:06:35.600489 1 tasks_processing.go:71] worker 4 working on validating_webhook_configurations task. I0320 11:06:35.600555 1 tasks_processing.go:71] worker 27 working on ceph_cluster task. I0320 11:06:35.600560 1 tasks_processing.go:71] worker 26 working on nodenetworkconfigurationpolicies task. I0320 11:06:35.600666 1 tasks_processing.go:71] worker 9 working on version task. I0320 11:06:35.600742 1 tasks_processing.go:71] worker 8 working on active_alerts task. I0320 11:06:35.600766 1 tasks_processing.go:71] worker 14 working on storage_classes task. I0320 11:06:35.600773 1 tasks_processing.go:71] worker 20 working on monitoring_persistent_volumes task. I0320 11:06:35.600786 1 tasks_processing.go:71] worker 21 working on dvo_metrics task. I0320 11:06:35.600885 1 tasks_processing.go:71] worker 18 working on ingress task. I0320 11:06:35.600948 1 tasks_processing.go:71] worker 22 working on sap_pods task. I0320 11:06:35.601080 1 tasks_processing.go:71] worker 25 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0320 11:06:35.601174 1 tasks_processing.go:71] worker 16 working on oauths task. I0320 11:06:35.601247 1 tasks_processing.go:71] worker 15 working on operators_pods_and_events task. I0320 11:06:35.600378 1 tasks_processing.go:69] worker 23 listening for tasks. I0320 11:06:35.601326 1 tasks_processing.go:71] worker 23 working on clusterroles task. I0320 11:06:35.601434 1 tasks_processing.go:71] worker 3 working on pod_network_connectivity_checks task. I0320 11:06:35.601091 1 tasks_processing.go:71] worker 13 working on node_features task. I0320 11:06:35.600743 1 tasks_processing.go:71] worker 29 working on certificate_signing_requests task. I0320 11:06:35.600385 1 tasks_processing.go:69] worker 24 listening for tasks. I0320 11:06:35.601706 1 tasks_processing.go:71] worker 24 working on ingress_certificates task. I0320 11:06:35.600388 1 tasks_processing.go:69] worker 28 listening for tasks. I0320 11:06:35.600344 1 tasks_processing.go:71] worker 0 working on operators task. I0320 11:06:35.601827 1 tasks_processing.go:71] worker 28 working on authentication task. I0320 11:06:35.600405 1 tasks_processing.go:69] worker 10 listening for tasks. I0320 11:06:35.601931 1 tasks_processing.go:71] worker 10 working on machines task. I0320 11:06:35.600408 1 tasks_processing.go:69] worker 5 listening for tasks. I0320 11:06:35.600549 1 tasks_processing.go:71] worker 11 working on sap_datahubs task. I0320 11:06:35.602166 1 tasks_processing.go:71] worker 5 working on silenced_alerts task. I0320 11:06:35.600753 1 tasks_processing.go:71] worker 6 working on storage_cluster task. I0320 11:06:35.600759 1 tasks_processing.go:71] worker 2 working on container_images task. I0320 11:06:35.601502 1 tasks_processing.go:71] worker 19 working on node_logs task. I0320 11:06:35.600378 1 tasks_processing.go:69] worker 17 listening for tasks. I0320 11:06:35.602506 1 tasks_processing.go:71] worker 17 working on nodes task. I0320 11:06:35.605001 1 tasks_processing.go:71] worker 31 working on openstack_dataplanenodesets task. I0320 11:06:35.605063 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 4.581766ms to process 0 records I0320 11:06:35.605083 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 4.501702ms to process 0 records I0320 11:06:35.605089 1 tasks_processing.go:71] worker 27 working on container_runtime_configs task. I0320 11:06:35.605781 1 tasks_processing.go:71] worker 22 working on olm_operators task. I0320 11:06:35.605791 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 4.774562ms to process 0 records I0320 11:06:35.605861 1 tasks_processing.go:71] worker 30 working on machine_sets task. I0320 11:06:35.605873 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 5.359007ms to process 0 records I0320 11:06:35.606058 1 tasks_processing.go:71] worker 26 working on qemu_kubevirt_launcher_logs task. I0320 11:06:35.606065 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 5.483529ms to process 0 records I0320 11:06:35.607277 1 tasks_processing.go:71] worker 1 working on openstack_controlplanes task. I0320 11:06:35.607392 1 recorder.go:75] Recording config/image with fingerprint=5e00a2237c65447d053825e02995b14f3a99cecfebe1433c6fd54ad5b4f57d13 I0320 11:06:35.607406 1 gather.go:177] gatherer "clusterconfig" function "image" took 6.770157ms to process 1 records I0320 11:06:35.608888 1 tasks_processing.go:71] worker 11 working on jaegers task. I0320 11:06:35.608914 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 6.775097ms to process 0 records I0320 11:06:35.609046 1 tasks_processing.go:71] worker 6 working on networks task. I0320 11:06:35.609059 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 6.713807ms to process 0 records E0320 11:06:35.609122 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0320 11:06:35.609133 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 7.586308ms to process 0 records I0320 11:06:35.609140 1 tasks_processing.go:71] worker 3 working on tsdb_status task. I0320 11:06:35.609429 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0320 11:06:35.609444 1 controller.go:368] The initial operator extension status is healthy I0320 11:06:35.609452 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0320 11:06:35.609457 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0320 11:06:35.609459 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0320 11:06:35.609482 1 controller.go:531] The operator is still being initialized I0320 11:06:35.609490 1 controller.go:554] The operator is healthy I0320 11:06:35.609559 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0320 11:06:35.609661 1 operator.go:328] started I0320 11:06:35.609572 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0320 11:06:35.609689 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer E0320 11:06:35.611695 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.244.24:9091: connect: connection refused I0320 11:06:35.611719 1 tasks_processing.go:71] worker 3 working on config_maps task. E0320 11:06:35.611730 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.244.24:9091: connect: connection refused I0320 11:06:35.611752 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 2.568366ms to process 0 records E0320 11:06:35.626827 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0320 11:06:35.626843 1 tasks_processing.go:71] worker 13 working on machine_config_pools task. I0320 11:06:35.626874 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 25.242631ms to process 0 records I0320 11:06:35.627215 1 tasks_processing.go:71] worker 18 working on lokistack task. I0320 11:06:35.627524 1 recorder.go:75] Recording config/ingress with fingerprint=e65bc8624e506642e21b4c316168ec606c5425da978c2067763e81ce99a5c112 I0320 11:06:35.627540 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 26.296127ms to process 1 records I0320 11:06:35.627548 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 26.480356ms to process 0 records I0320 11:06:35.627554 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 21.468672ms to process 0 records I0320 11:06:35.627562 1 tasks_processing.go:71] worker 30 working on service_accounts task. I0320 11:06:35.627704 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 20.410449ms to process 0 records I0320 11:06:35.627938 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 22.692135ms to process 0 records I0320 11:06:35.628065 1 tasks_processing.go:71] worker 31 working on sap_config task. I0320 11:06:35.628168 1 tasks_processing.go:71] worker 20 working on schedulers task. I0320 11:06:35.628381 1 tasks_processing.go:71] worker 1 working on metrics task. I0320 11:06:35.629002 1 tasks_processing.go:71] worker 10 working on pdbs task. E0320 11:06:35.629347 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0320 11:06:35.629405 1 gather.go:177] gatherer "clusterconfig" function "machines" took 27.023869ms to process 0 records I0320 11:06:35.629755 1 tasks_processing.go:71] worker 28 working on openshift_machine_api_events task. I0320 11:06:35.630525 1 recorder.go:75] Recording config/authentication with fingerprint=a4b662638c6106faaf803fbbc86f97fbc08beacd32386d78f843c4819af583ec I0320 11:06:35.630597 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 27.864915ms to process 1 records I0320 11:06:35.630637 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 25.519726ms to process 0 records I0320 11:06:35.630701 1 tasks_processing.go:71] worker 27 working on machine_healthchecks task. I0320 11:06:35.630572 1 gather_logs.go:145] no pods in namespace were found I0320 11:06:35.630832 1 tasks_processing.go:71] worker 26 working on image_registries task. I0320 11:06:35.630849 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 24.756483ms to process 0 records I0320 11:06:35.631233 1 tasks_processing.go:71] worker 6 working on infrastructures task. I0320 11:06:35.631367 1 recorder.go:75] Recording config/network with fingerprint=0c1da67edd076d3c27d02b421a6431a5a7db5f9405f517070f24ff7274e38d3d I0320 11:06:35.631417 1 gather.go:177] gatherer "clusterconfig" function "networks" took 21.834309ms to process 1 records E0320 11:06:35.632672 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.244.24:9091: connect: connection refused I0320 11:06:35.632704 1 tasks_processing.go:71] worker 1 working on openshift_logging task. E0320 11:06:35.632722 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.244.24:9091: connect: connection refused I0320 11:06:35.632766 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 4.254962ms to process 0 records I0320 11:06:35.635444 1 tasks_processing.go:71] worker 17 working on openstack_version task. I0320 11:06:35.635800 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0320 11:06:35.635854 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0320 11:06:35.635870 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0320 11:06:35.635878 1 controller.go:254] Source scaController *sca.Controller is not ready I0320 11:06:35.635882 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0320 11:06:35.635896 1 recorder.go:75] Recording config/node/ip-10-0-0-215.ec2.internal with fingerprint=c088ce1e3fe881a08ed3d723bc00c0726fe6f88f35883eec50692c868af92793 I0320 11:06:35.635915 1 controller.go:531] The operator is still being initialized I0320 11:06:35.635930 1 controller.go:554] The operator is healthy I0320 11:06:35.636007 1 recorder.go:75] Recording config/node/ip-10-0-1-143.ec2.internal with fingerprint=3c09d8942dd8e4c34c3f56de3206a9a2b43417c45302dc62907717b722e7be8c I0320 11:06:35.636137 1 recorder.go:75] Recording config/node/ip-10-0-2-149.ec2.internal with fingerprint=fb9c1270e76e3d3db071d219f08abfe951f11690cd0506a0a6f0c1e4f8bbffeb I0320 11:06:35.636148 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 32.923277ms to process 3 records I0320 11:06:35.636232 1 tasks_processing.go:71] worker 4 working on support_secret task. I0320 11:06:35.636306 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=772c2f8499a291f2b7b17a3518eedbae295a645554a2f70e4e52215430d478cb I0320 11:06:35.636403 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=312796e286ab690c4d719380c1a1ffa94381bb776dbf26004449c876968729bf I0320 11:06:35.636630 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=51d9338ae46d750069a5d34f4845d0daf96d1941e9ea791cdbd4462ec3915b25 I0320 11:06:35.636680 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=950399cec139bf2336a99de59b66b6623200f96426d51cdba4d013a8aa3a59ee I0320 11:06:35.636724 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=a62dbb784af9e19e868da2e287f1a2b33669dbee5e04d4efd4fc8bb89787fd89 I0320 11:06:35.636760 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=beee4d24f4d9ab48d6aa2ff6add4014bd70c65d4ab00188aa0faacdceb9937af I0320 11:06:35.636795 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=cf86450e61d1b176afb550624db94dbe65367ea739f8ed796c61b7cf452f2450 I0320 11:06:35.636826 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=4b894ab992799af31313a51ecc8edbe9f02730f55a118ffb4d13b5390b771e15 I0320 11:06:35.636858 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=6c74b3abc7330e79e447278cb0644eef7afb9f51acf84ab96208f38df19305e5 I0320 11:06:35.636892 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=802bfab5b581b9eab83674905ef397d1df52355824e1fc2d4c39ebbc93dd4ef2 I0320 11:06:35.636916 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=32a3252a419f7f657e0fb8402535751ce69b7522d585443599fc9370b37786ec I0320 11:06:35.636949 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=4e6acfb5ecc7afca89b46245dfa24315b4b03a1fbb6465ab5f4bd8629a1d63d0 I0320 11:06:35.636973 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=7c093fbbfb5a722a6342b41ec801f8a29d03e0584f12d2cbf8a7356bd34cef58 I0320 11:06:35.636981 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 35.112132ms to process 13 records I0320 11:06:35.638557 1 tasks_processing.go:71] worker 29 working on feature_gates task. I0320 11:06:35.638570 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 36.863077ms to process 0 records I0320 11:06:35.643640 1 tasks_processing.go:71] worker 15 working on mutating_webhook_configurations task. I0320 11:06:35.643652 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 42.365531ms to process 0 records I0320 11:06:35.644844 1 tasks_processing.go:71] worker 19 working on cost_management_metrics_configs task. I0320 11:06:35.644868 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 42.385043ms to process 0 records I0320 11:06:35.654154 1 tasks_processing.go:71] worker 12 working on install_plans task. I0320 11:06:35.654179 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0320 11:06:35.654191 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 53.782495ms to process 1 records I0320 11:06:35.655715 1 tasks_processing.go:71] worker 11 working on openstack_dataplanedeployments task. I0320 11:06:35.655730 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 46.807772ms to process 0 records I0320 11:06:35.655810 1 tasks_processing.go:71] worker 14 working on image_pruners task. I0320 11:06:35.655903 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=7a1ae141d0145ae1fbc877dabc65e5019ee7fa756134464eb75875ee6057c0b5 I0320 11:06:35.655964 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=429dc7df27c7ec0470a5e87b41877a50320b2fa983c0932bbc1dbb2585cc1358 I0320 11:06:35.655977 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 54.961383ms to process 2 records I0320 11:06:35.656062 1 tasks_processing.go:71] worker 16 working on machine_configs task. I0320 11:06:35.656358 1 recorder.go:75] Recording config/oauth with fingerprint=f94c0012bd01bc496908e7386ac8db71cb0f4c3b628bddd3f4f01110fd50317a I0320 11:06:35.656373 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 54.844104ms to process 1 records I0320 11:06:35.657481 1 tasks_processing.go:71] worker 13 working on crds task. I0320 11:06:35.657490 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 30.619478ms to process 0 records I0320 11:06:35.657860 1 tasks_processing.go:71] worker 1 working on proxies task. I0320 11:06:35.657898 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 25.147954ms to process 0 records I0320 11:06:35.657951 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 30.635795ms to process 0 records E0320 11:06:35.657984 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0320 11:06:35.657996 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 27.165603ms to process 0 records I0320 11:06:35.658059 1 tasks_processing.go:74] worker 18 stopped. I0320 11:06:35.658067 1 tasks_processing.go:71] worker 27 working on cluster_apiserver task. W0320 11:06:35.658447 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 11:06:35.658975 1 tasks_processing.go:74] worker 31 stopped. I0320 11:06:35.658999 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 30.865271ms to process 0 records I0320 11:06:35.664168 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0320 11:06:35.664178 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0320 11:06:35.664191 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0320 11:06:35.666964 1 tasks_processing.go:74] worker 26 stopped. I0320 11:06:35.667413 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=a419a9274f28eda54cc35c063cd19ea11bba95c48ffc42a8623138b6b8c60a39 I0320 11:06:35.667432 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 36.070201ms to process 1 records I0320 11:06:35.667594 1 tasks_processing.go:74] worker 6 stopped. I0320 11:06:35.668160 1 recorder.go:75] Recording config/infrastructure with fingerprint=c532cdcc6ca2d1f415c308183a6756d75b84a0df60293d767297f9b62e5da0a4 I0320 11:06:35.668174 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 36.341533ms to process 1 records I0320 11:06:35.672299 1 base_controller.go:82] Caches are synced for ConfigController I0320 11:06:35.672309 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0320 11:06:35.673680 1 tasks_processing.go:74] worker 10 stopped. I0320 11:06:35.673819 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=ce041908021ff7cf1fd733e9d3dab4dd959602df6c0c1f76a39e8f478f447038 I0320 11:06:35.673848 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=d4d8898d319b8fffadd8f746b3ea8e1cb4395ce129afc27611e64e553b063108 I0320 11:06:35.673865 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=481ab398058cea28fca46cfb5a680a6041761277f87ab4eae9c71a7d7a4d4f1a I0320 11:06:35.673883 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=7a49922242746e651713c080dc4af4e5bda571e585237e5b984958302df7120a I0320 11:06:35.673900 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=20751203a9b2ec1e432e6d921df881ccffd08261311ffe58940e615e7489a3b8 I0320 11:06:35.673912 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=30a66b97d8c7103aa2b0126a0c79aa64add50ebf55c69c1027b6649629681a45 I0320 11:06:35.673927 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=c849dd34fc85d791fadf2efa213230ec9706f3846b0541be0ca6e9da459ab0bc I0320 11:06:35.673944 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=f79a3ef03494ac6c5037b9c3b5168cf4b7b516ef9aaad7ccf72249a42a8eef9e I0320 11:06:35.673959 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=f2a66c918cc96708a392dc49085521ec5245abc6931a6f88483fbf277d93caae I0320 11:06:35.673975 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=2b11340b9b43756e6ffd457070571d11e0e7bfcc95b64322b1ca6dfb00185c1e I0320 11:06:35.673989 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=2039ca3da327906fefaae92a2bd7c03bf285b6548808c2a7e41c54149fcb6edb I0320 11:06:35.673998 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 44.603744ms to process 11 records I0320 11:06:35.674070 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=da046e7aa069ddf756a999211d9cf6bde2f0eb6315eff32794eface17f97dbf3 I0320 11:06:35.674092 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 45.582943ms to process 1 records I0320 11:06:35.674103 1 tasks_processing.go:74] worker 20 stopped. I0320 11:06:35.682463 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 37.535365ms to process 0 records I0320 11:06:35.682397 1 tasks_processing.go:74] worker 19 stopped. I0320 11:06:35.682766 1 tasks_processing.go:74] worker 17 stopped. I0320 11:06:35.682795 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 47.288066ms to process 0 records I0320 11:06:35.682843 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 27.081894ms to process 0 records I0320 11:06:35.682886 1 tasks_processing.go:74] worker 11 stopped. I0320 11:06:35.685738 1 tasks_processing.go:74] worker 28 stopped. I0320 11:06:35.685750 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 55.913938ms to process 0 records I0320 11:06:35.688147 1 tasks_processing.go:74] worker 4 stopped. E0320 11:06:35.688160 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0320 11:06:35.688170 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 51.901859ms to process 0 records I0320 11:06:35.688588 1 tasks_processing.go:74] worker 29 stopped. I0320 11:06:35.688740 1 recorder.go:75] Recording config/featuregate with fingerprint=33aa58605287662929abb35b2e27e5da5a3ecb5e86e293db9b39cdae80e5bb7a I0320 11:06:35.688756 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 50.014825ms to process 1 records I0320 11:06:35.689145 1 tasks_processing.go:74] worker 23 stopped. I0320 11:06:35.689340 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=f7b96081184820956b27d43f14fb66107a0f0bd8ed31fad9a9f3dc712445981b I0320 11:06:35.689441 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=3db22ba4a9ff3812abc45075ed7380feb09f8fdd39bebdff29bec3422b5d2f0e I0320 11:06:35.689451 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 87.798211ms to process 2 records I0320 11:06:35.689753 1 tasks_processing.go:74] worker 16 stopped. E0320 11:06:35.689762 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0320 11:06:35.689789 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0320 11:06:35.689801 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 33.672532ms to process 1 records I0320 11:06:35.699610 1 tasks_processing.go:74] worker 2 stopped. I0320 11:06:35.700894 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 11:06:35.702625 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-5b5d9b4544-q84lt with fingerprint=8e702c4233988d94442454936b7c6f17d5d9efa885244bc4b3138bec47e6e834 I0320 11:06:35.703124 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-kchcs with fingerprint=24c55ee5d429f61fa783196a8f81d071b921e6a7f358ffcb5b5b1462b119a5d6 I0320 11:06:35.703657 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-xcjt2 with fingerprint=3754941785a030717729d451110a88e75ba97137f4dcf68d3f4abfd3e2fff447 I0320 11:06:35.704038 1 recorder.go:75] Recording config/running_containers with fingerprint=158efdd50a6caf24ba901cc1ecc3ea6160ee5f7b66ada13c40ad51735fed630a I0320 11:06:35.704055 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 97.223337ms to process 4 records I0320 11:06:35.704141 1 tasks_processing.go:74] worker 14 stopped. I0320 11:06:35.704175 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=91193f674ec69b6cbcf6ef422b0779b97920f5f3dce62a941773300f9e9e48b3 I0320 11:06:35.704183 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 45.264167ms to process 1 records I0320 11:06:35.704250 1 recorder.go:75] Recording config/proxy with fingerprint=641e0df56b46dbf3dd8b829128ac4e70fc80f0fbd34ee306dc4dadcd1af77e39 I0320 11:06:35.704259 1 tasks_processing.go:74] worker 1 stopped. I0320 11:06:35.704262 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 43.216877ms to process 1 records I0320 11:06:35.704341 1 tasks_processing.go:74] worker 27 stopped. I0320 11:06:35.704415 1 recorder.go:75] Recording config/apiserver with fingerprint=178f6b1b55d33eb7342d9fc51934cafaad84bb9b446691d61cbba6ad310d5806 I0320 11:06:35.704429 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 43.246881ms to process 1 records I0320 11:06:35.704485 1 recorder.go:75] Recording config/olm_operators with fingerprint=cbbf46ff060a45ab9e6baf433e5e5fcbeda37ef7dccd60eb8767bdec4af04f01 I0320 11:06:35.704495 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 95.863484ms to process 1 records I0320 11:06:35.704502 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 103.993735ms to process 0 records I0320 11:06:35.704512 1 tasks_processing.go:74] worker 7 stopped. I0320 11:06:35.704520 1 tasks_processing.go:74] worker 22 stopped. I0320 11:06:35.705070 1 tasks_processing.go:74] worker 15 stopped. I0320 11:06:35.705225 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=ce226550f00da1483e4deb92e5f2d13ebff8c7abfdbc1b98e89508702b7032e6 I0320 11:06:35.705272 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=215d6c099e581608a05e3d16ee14c2b358b21100d68524666a63f0b26b99fae9 I0320 11:06:35.705313 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=008b5b796de4205e7b296bc7dab405176a2fc81de4d28a65891dfeb0f00cf2f2 I0320 11:06:35.705324 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 61.410224ms to process 3 records I0320 11:06:35.708959 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0320 11:06:35.709879 1 base_controller.go:82] Caches are synced for LoggingSyncer I0320 11:06:35.709890 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0320 11:06:35.712870 1 tasks_processing.go:74] worker 13 stopped. I0320 11:06:35.713305 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=44fe7fc8e5d67074abe8ba7c246d805c2aa69a6f272f3e0a4a12b9bbc3fdbdd3 I0320 11:06:35.713508 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=a576230ce1fe87a26a38e9ba82ccd5bd2e899e55adaf6c3d512b9619729af20f I0320 11:06:35.713518 1 gather.go:177] gatherer "clusterconfig" function "crds" took 55.336918ms to process 2 records I0320 11:06:35.713612 1 tasks_processing.go:74] worker 9 stopped. I0320 11:06:35.713740 1 recorder.go:75] Recording config/version with fingerprint=de167cba6939b3b69edf7c0017422fd080883c5d1ae081105e2ddfd400f590e5 I0320 11:06:35.713752 1 recorder.go:75] Recording config/id with fingerprint=49fb132acd73404ed15552b9037a263d06c14a7da256de1a9c53330b03b706ce I0320 11:06:35.713769 1 gather.go:177] gatherer "clusterconfig" function "version" took 112.510217ms to process 2 records I0320 11:06:35.718260 1 tasks_processing.go:74] worker 25 stopped. I0320 11:06:35.718287 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 117.119512ms to process 0 records I0320 11:06:35.721535 1 tasks_processing.go:74] worker 3 stopped. E0320 11:06:35.721601 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0320 11:06:35.721632 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0320 11:06:35.721667 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0320 11:06:35.721706 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=c8a9805405aae7d04b7f03c09c51097942070877009b6daebd5573ef8127446e I0320 11:06:35.721799 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0320 11:06:35.721832 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0320 11:06:35.721859 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=3b70b9787487b548204d806ebebf90b0c2d460145e34cb33c0d8d3a7a011d907 I0320 11:06:35.721902 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0320 11:06:35.721986 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0320 11:06:35.722053 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0320 11:06:35.722092 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 109.804765ms to process 7 records E0320 11:06:35.722137 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0320 11:06:35.722164 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2p5dfcheert4iiujdq8kqp8439v2eqd2-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2p5dfcheert4iiujdq8kqp8439v2eqd2-primary-cert-bundle-secret" not found I0320 11:06:35.722241 1 tasks_processing.go:74] worker 24 stopped. I0320 11:06:35.722265 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=56f7e48a603448f9833f0e15b3cad12c1c9c6bf1187c86a5c9cededbef8eb708 I0320 11:06:35.722308 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 120.419371ms to process 1 records I0320 11:06:35.771442 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0320 11:06:35.776247 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0320 11:06:35.796824 1 cluster_transfer.go:102] no available accepted cluster transfer I0320 11:06:35.796838 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0320 11:06:35.880886 1 sca.go:266] etc-pki-entitlement secret successfully created I0320 11:06:35.880908 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated W0320 11:06:36.658521 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 11:06:36.663217 1 tasks_processing.go:74] worker 5 stopped. I0320 11:06:36.663238 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0320 11:06:36.663250 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.061028419s to process 1 records I0320 11:06:36.663297 1 tasks_processing.go:74] worker 8 stopped. I0320 11:06:36.663331 1 recorder.go:75] Recording config/alerts with fingerprint=1d9cce20ce6bd8e859d39972831a1a74b55001560ef9c5883fcf58619d8def06 I0320 11:06:36.663345 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.062519121s to process 1 records W0320 11:06:37.658541 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 11:06:38.117721 1 tasks_processing.go:74] worker 0 stopped. I0320 11:06:38.117852 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=abb63c642405cf2839d9a87695ac80503b4b48f191123365250db70c7059b563 I0320 11:06:38.117899 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0320 11:06:38.117928 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=2d0177d3985a8b93d98cc7c60bd29a86c65db5ae6ee81e4c53adf3ab78115c10 I0320 11:06:38.117938 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0320 11:06:38.117959 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=ab01646257315a238c31910c86b58cb31d2039e8b096e1674884886295aa220e I0320 11:06:38.117975 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0320 11:06:38.117997 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=1e959934ebc8c792591f7e9a54c8ee1c7102f958278bda5717485aab8c0956d0 I0320 11:06:38.118052 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=191f21c8c235dc9ce8cc41c7f4f4a5559bfc24c693ac7b04b6db9c870b14badc I0320 11:06:38.118076 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=006e33ef56d3e6ec7e97d357d117c691ebea5bf03534421d4c11907ad2e951eb I0320 11:06:38.118106 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=3df7cdfb312348b3f53723a0ada3f92ad80cedee725d4199773f5c5cd6be99e5 I0320 11:06:38.118405 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=eee3bbfd86d389358ec1c9e81156e9935268fd6cef441af6151a870c28b3d67a I0320 11:06:38.118426 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=e70ca4dfcd2e42fdb04cde0c350c0992d760beb126d394708041e0d53f3d618d I0320 11:06:38.118435 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0320 11:06:38.118451 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=6a48fa922188a02e65408adc521452c14beb71cb832b79d9826cfa8d91a9984f I0320 11:06:38.118464 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0320 11:06:38.118480 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=d0af31a0fa2c2c985fee5ff5998a7227b10fa6a0fd7d4d102098e000594a48a2 I0320 11:06:38.118488 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0320 11:06:38.118507 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=0b61cb54c6822363ac83e37c063e1a54857a1aa863cbaea1089613f66b1ecf56 I0320 11:06:38.118516 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0320 11:06:38.118537 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=bb332d29ca6f6e3916efeceef3f4330105a5bddb0eb495fa73e85fd886c0db1d I0320 11:06:38.118640 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=8cc3c2a488aea89d0a1fe5f965b96adae5acfda9624b570d415af8bfccc54cba I0320 11:06:38.118649 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0320 11:06:38.118662 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0320 11:06:38.118684 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0320 11:06:38.118706 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=4408a66b2245cf539ca0ab6119e58ad2630e63f4622cd1b591797c26f0b7d1d2 I0320 11:06:38.118729 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=61f987c8849afc25a0e548dc027a9d32de81983e0ea5d76997d4d18f6dd3f38f I0320 11:06:38.118737 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0320 11:06:38.118752 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=ec56a33da6a4dfe9939875e788b95c698df4f0c3060e05ae76d3131dc9561e2b I0320 11:06:38.118762 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0320 11:06:38.118780 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=101b33ce4ba296c81b412fc99e215ee26dc15a237964add7123003943a758f61 I0320 11:06:38.118789 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0320 11:06:38.118802 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=5b44641ec83a8c4e22951a3c9c19c2921e93639a3c36af6dda359c811633e85c I0320 11:06:38.118817 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=b59eb8a311127c67fcd5effb4c9e1f3e50c0f23d172087ada473bedba328e63d I0320 11:06:38.118832 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=e345ef661413cd263cb2089a0886830dabb499cf1e064bdff403176de5bea7d3 I0320 11:06:38.118851 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=4d40da6b3ee0b675be0cd3d467018b1f2359a5363ebb46d803500d73237a3158 I0320 11:06:38.118861 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0320 11:06:38.118883 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=b3f11f46505b5772d1b72f4cd8e5d0790e2833c25129c7e5faee0866f5db993f I0320 11:06:38.118899 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0320 11:06:38.118908 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0320 11:06:38.118914 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.515927366s to process 39 records W0320 11:06:38.658130 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0320 11:06:39.657955 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0320 11:06:40.657874 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0320 11:06:40.657920 1 tasks_processing.go:74] worker 21 stopped. E0320 11:06:40.657949 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0320 11:06:40.657970 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0320 11:06:40.657981 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.05706587s to process 1 records I0320 11:06:48.529207 1 tasks_processing.go:74] worker 12 stopped. I0320 11:06:48.529260 1 recorder.go:75] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0320 11:06:48.529276 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.875029934s to process 1 records I0320 11:06:49.234215 1 tasks_processing.go:74] worker 30 stopped. I0320 11:06:49.234494 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=168fffbb2ffe3e4223178b9929987b11ee2f64fe6f0ebe7873bbab778d18dbe6 I0320 11:06:49.234511 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.606636173s to process 1 records E0320 11:06:49.234574 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.634s with: function \"pod_network_connectivity_checks\" failed with an error, function \"tsdb_status\" failed with an error, function \"machines\" failed with an error, function \"metrics\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0320 11:06:49.235664 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "tsdb_status" failed with an error, function "machines" failed with an error, function "metrics" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0320 11:06:49.235675 1 periodic.go:216] Running workloads gatherer I0320 11:06:49.235689 1 tasks_processing.go:45] number of workers: 2 I0320 11:06:49.235698 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 11:06:49.235702 1 tasks_processing.go:71] worker 1 working on workload_info task. I0320 11:06:49.235712 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 11:06:49.235733 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0320 11:06:49.242196 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-149.ec2.internal... I0320 11:06:49.242207 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-143.ec2.internal... I0320 11:06:49.242198 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-215.ec2.internal... I0320 11:06:49.262968 1 tasks_processing.go:74] worker 0 stopped. I0320 11:06:49.262994 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 27.208631ms to process 0 records I0320 11:06:53.596760 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 4s I0320 11:06:53.621843 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 25s for image data I0320 11:06:53.629810 1 gather_workloads_info.go:387] No image sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6 (9ms) I0320 11:06:53.639506 1 gather_workloads_info.go:387] No image sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319 (10ms) I0320 11:06:53.648156 1 gather_workloads_info.go:387] No image sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575 (9ms) I0320 11:06:53.656225 1 gather_workloads_info.go:387] No image sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7 (8ms) I0320 11:06:53.664197 1 gather_workloads_info.go:387] No image sha256:3666bcb850e1518aec86d322af52e95f725602ad0ab770d2b1e5b39b6dbde0b5 (8ms) I0320 11:06:53.672089 1 gather_workloads_info.go:387] No image sha256:3e889ea85c9c4748c548b84ae135a8adc88f42026ce8073538ca2fe7d89b8004 (8ms) I0320 11:06:53.680896 1 gather_workloads_info.go:398] Found image sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd (9ms) I0320 11:06:53.689322 1 gather_workloads_info.go:387] No image sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6 (8ms) I0320 11:06:53.697639 1 gather_workloads_info.go:398] Found image sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814 (8ms) I0320 11:06:53.705480 1 gather_workloads_info.go:387] No image sha256:6521242b8805e70936f8e49c7d50a319adaeb73b4e5dec4693b0f93a34e397c5 (8ms) I0320 11:06:53.729390 1 gather_workloads_info.go:387] No image sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c (24ms) I0320 11:06:53.831479 1 gather_workloads_info.go:387] No image sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df (102ms) I0320 11:06:53.930626 1 gather_workloads_info.go:387] No image sha256:7db22f56cf9d223b1437e8cba3aa9e04e8ae8b973e23e85133bf1d3d8d66f15e (99ms) I0320 11:06:54.030441 1 gather_workloads_info.go:387] No image sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044 (100ms) I0320 11:06:54.130566 1 gather_workloads_info.go:387] No image sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564 (100ms) I0320 11:06:54.230443 1 gather_workloads_info.go:387] No image sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc (100ms) I0320 11:06:54.330487 1 gather_workloads_info.go:387] No image sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f (100ms) I0320 11:06:54.429937 1 gather_workloads_info.go:387] No image sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e (99ms) I0320 11:06:54.531556 1 gather_workloads_info.go:387] No image sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb (102ms) I0320 11:06:54.631322 1 gather_workloads_info.go:387] No image sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d (100ms) I0320 11:06:54.730843 1 gather_workloads_info.go:387] No image sha256:60648d27017709a420e43353f9a83d7c466017b3b771200672a4dbc3c3f7a3ae (99ms) I0320 11:06:54.830556 1 gather_workloads_info.go:387] No image sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79 (100ms) I0320 11:06:54.930553 1 gather_workloads_info.go:387] No image sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43 (100ms) I0320 11:06:55.030373 1 gather_workloads_info.go:387] No image sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809 (100ms) I0320 11:06:55.130986 1 gather_workloads_info.go:387] No image sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790 (101ms) I0320 11:06:55.233654 1 gather_workloads_info.go:387] No image sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498 (103ms) I0320 11:06:55.331615 1 gather_workloads_info.go:387] No image sha256:07f14370b57046dc2ac49be933bb2248aaf5943e8fb56ce9703eaa7c673b999f (98ms) I0320 11:06:55.430441 1 gather_workloads_info.go:387] No image sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf (99ms) I0320 11:06:55.531369 1 gather_workloads_info.go:387] No image sha256:188e4e2d697e2ca3c7c16772c0ab12253f00bd8433d8691ed7058fe7bafbbbf4 (101ms) I0320 11:06:55.631260 1 gather_workloads_info.go:387] No image sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635 (100ms) I0320 11:06:55.730869 1 gather_workloads_info.go:387] No image sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd (100ms) I0320 11:06:55.830706 1 gather_workloads_info.go:387] No image sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211 (100ms) I0320 11:06:55.930764 1 gather_workloads_info.go:387] No image sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a (100ms) I0320 11:06:56.030693 1 gather_workloads_info.go:387] No image sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083 (100ms) I0320 11:06:56.130920 1 gather_workloads_info.go:387] No image sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4 (100ms) I0320 11:06:56.230288 1 gather_workloads_info.go:387] No image sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958 (99ms) I0320 11:06:56.330706 1 gather_workloads_info.go:387] No image sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648 (100ms) I0320 11:06:56.430498 1 gather_workloads_info.go:387] No image sha256:50b7c5185c4a6e63145753415c92723c59136d14b951ea0f4f06739b00893329 (100ms) I0320 11:06:56.530359 1 gather_workloads_info.go:387] No image sha256:a9dc28a6962d195dcce0c76a544aad5f784dd90b98282c169d7b94c870a85f8b (100ms) I0320 11:06:56.631661 1 gather_workloads_info.go:387] No image sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea (101ms) I0320 11:06:56.731404 1 gather_workloads_info.go:387] No image sha256:0b2cc6aadb1abc1d35a55da0c9884eb9426d7ff7a97174daeb09bb1e1a23cfae (100ms) I0320 11:06:56.831640 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (100ms) I0320 11:06:56.930695 1 gather_workloads_info.go:387] No image sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329 (99ms) I0320 11:06:57.030836 1 gather_workloads_info.go:387] No image sha256:3e772c41ef85aaccf63c409ee72b43d632b36d4cc66a3790835dffdcf9b7433e (100ms) I0320 11:06:57.131822 1 gather_workloads_info.go:387] No image sha256:11de6fe1741f5dd485e59ee2e703868badda81de0a179513be70b6ac9fc4655f (101ms) I0320 11:06:57.230689 1 gather_workloads_info.go:387] No image sha256:3d68a9523f4c5c22a48dfa151bbfde78b2f360697afe3b9d8bfb53e8c359bf95 (99ms) I0320 11:06:57.330967 1 gather_workloads_info.go:387] No image sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e (100ms) I0320 11:06:57.430307 1 gather_workloads_info.go:387] No image sha256:12e6985890d3c549f7cc9881892793b27277f72e2739cb910aaf9f4dd7215df1 (99ms) I0320 11:06:57.531448 1 gather_workloads_info.go:387] No image sha256:ab32e70fb149ae7470c941ca73bcfe9eba540bd5058a3d697a3ddf96597d0132 (101ms) I0320 11:06:57.531483 1 tasks_processing.go:74] worker 1 stopped. I0320 11:06:57.531820 1 recorder.go:75] Recording config/workload_info with fingerprint=66e728d8c9e4992562e0719bd6ed342e91bea4272b4ae0e27d0a89e5e2017c4c I0320 11:06:57.531846 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.295774499s to process 1 records I0320 11:06:57.531865 1 periodic.go:263] Periodic gather workloads completed in 8.296s I0320 11:06:57.531872 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0320 11:06:57.531877 1 periodic.go:216] Running conditional gatherer I0320 11:06:57.540626 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.5/gathering_rules I0320 11:06:57.709297 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer E0320 11:06:57.711916 1 conditional_gatherer.go:324] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp 172.30.244.24:9091: connect: connection refused I0320 11:06:57.711988 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0320 11:06:57.718158 1 conditional_gatherer.go:392] cluster version is '4.21.5' E0320 11:06:57.718175 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:06:57.718181 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:06:57.718184 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:06:57.718187 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:06:57.718190 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:06:57.718193 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:06:57.718196 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:06:57.718198 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:06:57.718202 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 11:06:57.718205 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0320 11:06:57.718220 1 tasks_processing.go:45] number of workers: 3 I0320 11:06:57.718229 1 tasks_processing.go:69] worker 2 listening for tasks. I0320 11:06:57.718233 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0320 11:06:57.718245 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 11:06:57.718259 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 11:06:57.718263 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0320 11:06:57.718271 1 tasks_processing.go:74] worker 1 stopped. I0320 11:06:57.718275 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0320 11:06:57.718306 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=9fafdcbf373f1f367f1d69551ea59114592cce1681189be0f3dde774e0530661 I0320 11:06:57.718319 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.082µs to process 1 records I0320 11:06:57.718355 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0320 11:06:57.718363 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.75µs to process 1 records I0320 11:06:57.718369 1 tasks_processing.go:74] worker 0 stopped. I0320 11:06:57.718551 1 tasks_processing.go:74] worker 2 stopped. I0320 11:06:57.718563 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 266.982µs to process 0 records I0320 11:06:57.718578 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0320 11:06:57.718595 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0320 11:06:57.746998 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=d12b25dc49a72dc13cfc0889d8beda4a1741c0d9bdd02f37f795ea543cd3351a I0320 11:06:57.747162 1 diskrecorder.go:70] Writing 113 records to /var/lib/insights-operator/insights-2026-03-20-110657.tar.gz I0320 11:06:57.754343 1 diskrecorder.go:51] Wrote 113 records to disk in 7ms I0320 11:06:57.754372 1 periodic.go:285] Gathering cluster info every 2h0m0s I0320 11:06:57.754385 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0320 11:08:05.601301 1 diskrecorder.go:223] Found files to send: insights-2026-03-20-110657.tar.gz I0320 11:08:05.601327 1 insightsuploader.go:150] Checking archives to upload periodically every 15m29.496177346s I0320 11:08:05.601335 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0320 11:08:05.615013 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0320 11:08:05.845801 1 requests.go:87] Successfully reported id=2026-03-20T11:08:05Z x-rh-insights-request-id=e081e15d4cb549a7bb447e4cd456bea1, wrote=52873 I0320 11:08:05.845854 1 insightsuploader.go:187] Uploaded report successfully in 244.511907ms I0320 11:08:05.845877 1 controller.go:129] Initializing last reported time to 2026-03-20T11:08:05Z I0320 11:08:05.845909 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0320 11:08:05.845918 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0320 11:08:05.845925 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0320 11:08:05.851850 1 controller.go:554] The operator is healthy I0320 11:08:35.618690 1 controller.go:554] The operator is healthy I0320 11:09:07.234345 1 insightsreport.go:137] Pulling report from smart-proxy I0320 11:09:07.234374 1 insightsreport.go:149] Retrieving report I0320 11:09:07.241221 1 requests.go:110] Retrieving report for cluster: fade2ce1-cbab-4de1-8e87-9348a8e820ac I0320 11:09:07.241234 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/fade2ce1-cbab-4de1-8e87-9348a8e820ac/reports I0320 11:09:07.244333 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/fade2ce1-cbab-4de1-8e87-9348a8e820ac/reports I0320 11:09:07.423938 1 insightsreport.go:184] Report retrieved I0320 11:09:07.452259 1 insightsreport.go:239] Report retrieved correctly I0320 11:10:35.616815 1 controller.go:554] The operator is healthy I0320 11:11:35.591264 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 11:11:35.597129 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 11:11:35.597146 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 11:11:35.601696 1 secretconfigobserver.go:119] support secret does not exist I0320 11:12:35.617089 1 controller.go:554] The operator is healthy I0320 11:14:35.620144 1 controller.go:554] The operator is healthy I0320 11:16:35.592800 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 11:16:35.597787 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 11:16:35.601875 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 11:16:35.606813 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 11:16:35.606830 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 11:16:35.611459 1 secretconfigobserver.go:119] support secret does not exist I0320 11:16:35.616705 1 controller.go:554] The operator is healthy I0320 11:18:35.620498 1 controller.go:554] The operator is healthy I0320 11:20:35.619823 1 controller.go:554] The operator is healthy I0320 11:21:35.612014 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 11:21:35.621495 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 11:21:35.621520 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 11:21:35.626513 1 secretconfigobserver.go:119] support secret does not exist I0320 11:22:35.620117 1 controller.go:554] The operator is healthy I0320 11:23:40.343739 1 insightsuploader.go:146] Nothing to report since 2026-03-20T11:08:05Z I0320 11:24:35.617261 1 controller.go:554] The operator is healthy I0320 11:26:35.589499 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 11:26:35.616703 1 controller.go:554] The operator is healthy I0320 11:26:35.626569 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 11:26:35.631575 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 11:26:35.631595 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 11:26:35.636197 1 secretconfigobserver.go:119] support secret does not exist I0320 11:28:35.616576 1 controller.go:554] The operator is healthy I0320 11:30:35.616855 1 controller.go:554] The operator is healthy I0320 11:31:35.637231 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 11:31:35.646926 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 11:31:35.646944 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 11:31:35.651221 1 secretconfigobserver.go:119] support secret does not exist I0320 11:32:35.616890 1 controller.go:554] The operator is healthy I0320 11:34:35.620013 1 controller.go:554] The operator is healthy I0320 11:36:35.591085 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 11:36:35.616330 1 controller.go:554] The operator is healthy I0320 11:36:35.652278 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 11:36:35.657674 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 11:36:35.657692 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 11:36:35.662351 1 secretconfigobserver.go:119] support secret does not exist I0320 11:38:35.616906 1 controller.go:554] The operator is healthy I0320 11:39:51.378203 1 insightsuploader.go:146] Nothing to report since 2026-03-20T11:08:05Z I0320 11:40:35.616370 1 controller.go:554] The operator is healthy I0320 11:41:35.662636 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 11:41:35.668147 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 11:41:35.668166 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 11:41:35.672770 1 secretconfigobserver.go:119] support secret does not exist