I0320 12:41:44.957474 1 cmd.go:253] Using service-serving-cert provided certificates I0320 12:41:44.958247 1 observer_polling.go:159] Starting file observer I0320 12:41:45.802271 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0320 12:41:45.802570 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0320 12:41:45.803130 1 secure_serving.go:57] Forcing use of http/1.1 only W0320 12:41:45.803149 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0320 12:41:45.803155 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0320 12:41:45.803159 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0320 12:41:45.803163 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0320 12:41:45.803165 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0320 12:41:45.803167 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0320 12:41:45.803519 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0320 12:41:45.807423 1 secure_serving.go:213] Serving securely on [::]:8443 I0320 12:41:45.807471 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0320 12:41:45.807479 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0320 12:41:45.807514 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0320 12:41:45.807626 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" I0320 12:41:45.807738 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0320 12:41:45.807746 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0320 12:41:45.807763 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0320 12:41:45.807769 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0320 12:41:45.808396 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"6c7e1aeb-0176-4bee-a645-795b44f0d4ae", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0320 12:41:45.808342 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] W0320 12:41:45.816259 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0320 12:41:45.816295 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0320 12:41:45.816410 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0320 12:41:45.823293 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 12:41:45.823312 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0320 12:41:45.826702 1 secretconfigobserver.go:119] support secret does not exist I0320 12:41:45.830545 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 12:41:45.833818 1 secretconfigobserver.go:119] support secret does not exist I0320 12:41:45.836174 1 recorder.go:176] Pruning old reports every 6h6m51s, max age is 288h0m0s I0320 12:41:45.840218 1 periodic.go:216] Running clusterconfig gatherer I0320 12:41:45.840232 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0320 12:41:45.840252 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0320 12:41:45.840257 1 tasks_processing.go:45] number of workers: 32 I0320 12:41:45.840278 1 tasks_processing.go:69] worker 3 listening for tasks. I0320 12:41:45.840289 1 tasks_processing.go:71] worker 3 working on schedulers task. I0320 12:41:45.840299 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 12:41:45.840302 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0320 12:41:45.840314 1 insightsreport.go:296] Starting report retriever I0320 12:41:45.840316 1 tasks_processing.go:69] worker 2 listening for tasks. I0320 12:41:45.840322 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0320 12:41:45.840325 1 tasks_processing.go:69] worker 17 listening for tasks. I0320 12:41:45.840307 1 tasks_processing.go:71] worker 1 working on support_secret task. I0320 12:41:45.840327 1 tasks_processing.go:69] worker 24 listening for tasks. I0320 12:41:45.840336 1 tasks_processing.go:69] worker 10 listening for tasks. I0320 12:41:45.840339 1 tasks_processing.go:69] worker 18 listening for tasks. I0320 12:41:45.840344 1 tasks_processing.go:69] worker 21 listening for tasks. I0320 12:41:45.840345 1 tasks_processing.go:69] worker 4 listening for tasks. I0320 12:41:45.840351 1 tasks_processing.go:69] worker 19 listening for tasks. I0320 12:41:45.840352 1 tasks_processing.go:69] worker 5 listening for tasks. I0320 12:41:45.840357 1 tasks_processing.go:71] worker 19 working on node_features task. I0320 12:41:45.840362 1 tasks_processing.go:69] worker 6 listening for tasks. I0320 12:41:45.840362 1 tasks_processing.go:71] worker 17 working on nodenetworkconfigurationpolicies task. I0320 12:41:45.840370 1 tasks_processing.go:71] worker 10 working on machine_autoscalers task. I0320 12:41:45.840376 1 tasks_processing.go:69] worker 22 listening for tasks. I0320 12:41:45.840386 1 tasks_processing.go:69] worker 23 listening for tasks. I0320 12:41:45.840328 1 tasks_processing.go:71] worker 2 working on metrics task. I0320 12:41:45.840543 1 tasks_processing.go:69] worker 25 listening for tasks. I0320 12:41:45.840560 1 tasks_processing.go:69] worker 26 listening for tasks. I0320 12:41:45.840570 1 tasks_processing.go:69] worker 27 listening for tasks. I0320 12:41:45.840574 1 tasks_processing.go:71] worker 5 working on sap_pods task. I0320 12:41:45.840585 1 tasks_processing.go:69] worker 28 listening for tasks. I0320 12:41:45.840591 1 tasks_processing.go:71] worker 28 working on oauths task. I0320 12:41:45.840596 1 tasks_processing.go:71] worker 21 working on sap_datahubs task. I0320 12:41:45.840649 1 tasks_processing.go:69] worker 11 listening for tasks. I0320 12:41:45.840664 1 tasks_processing.go:69] worker 12 listening for tasks. I0320 12:41:45.840672 1 tasks_processing.go:69] worker 13 listening for tasks. I0320 12:41:45.840682 1 tasks_processing.go:69] worker 14 listening for tasks. I0320 12:41:45.840691 1 tasks_processing.go:69] worker 15 listening for tasks. I0320 12:41:45.840699 1 tasks_processing.go:69] worker 16 listening for tasks. I0320 12:41:45.840330 1 tasks_processing.go:69] worker 20 listening for tasks. I0320 12:41:45.840569 1 tasks_processing.go:71] worker 26 working on infrastructures task. I0320 12:41:45.840744 1 tasks_processing.go:69] worker 29 listening for tasks. I0320 12:41:45.840758 1 tasks_processing.go:69] worker 30 listening for tasks. I0320 12:41:45.840357 1 tasks_processing.go:71] worker 4 working on ceph_cluster task. I0320 12:41:45.840767 1 tasks_processing.go:69] worker 31 listening for tasks. I0320 12:41:45.840775 1 tasks_processing.go:71] worker 31 working on openstack_version task. I0320 12:41:45.840832 1 tasks_processing.go:69] worker 7 listening for tasks. I0320 12:41:45.840848 1 tasks_processing.go:71] worker 7 working on pod_network_connectivity_checks task. I0320 12:41:45.840290 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 12:41:45.840937 1 tasks_processing.go:71] worker 14 working on jaegers task. I0320 12:41:45.840945 1 tasks_processing.go:71] worker 15 working on operators_pods_and_events task. I0320 12:41:45.841000 1 tasks_processing.go:71] worker 20 working on tsdb_status task. I0320 12:41:45.840366 1 tasks_processing.go:71] worker 24 working on openstack_controlplanes task. I0320 12:41:45.840591 1 tasks_processing.go:71] worker 18 working on validating_webhook_configurations task. I0320 12:41:45.841177 1 tasks_processing.go:71] worker 13 working on openstack_dataplanenodesets task. I0320 12:41:45.841209 1 tasks_processing.go:71] worker 16 working on container_runtime_configs task. I0320 12:41:45.841027 1 tasks_processing.go:69] worker 9 listening for tasks. I0320 12:41:45.841282 1 tasks_processing.go:71] worker 9 working on crds task. I0320 12:41:45.841077 1 tasks_processing.go:71] worker 29 working on config_maps task. I0320 12:41:45.841038 1 tasks_processing.go:69] worker 8 listening for tasks. I0320 12:41:45.841090 1 tasks_processing.go:71] worker 6 working on storage_classes task. I0320 12:41:45.841085 1 tasks_processing.go:71] worker 22 working on install_plans task. I0320 12:41:45.841101 1 tasks_processing.go:71] worker 30 working on node_logs task. I0320 12:41:45.840577 1 tasks_processing.go:71] worker 27 working on container_images task. I0320 12:41:45.841108 1 tasks_processing.go:71] worker 23 working on openstack_dataplanedeployments task. I0320 12:41:45.841120 1 tasks_processing.go:71] worker 11 working on dvo_metrics task. I0320 12:41:45.841160 1 tasks_processing.go:71] worker 12 working on networks task. I0320 12:41:45.841170 1 tasks_processing.go:71] worker 0 working on machines task. I0320 12:41:45.841435 1 tasks_processing.go:71] worker 8 working on machine_config_pools task. I0320 12:41:45.841114 1 tasks_processing.go:71] worker 25 working on authentication task. I0320 12:41:45.848438 1 tasks_processing.go:71] worker 4 working on cluster_apiserver task. I0320 12:41:45.849212 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 7.664861ms to process 0 records I0320 12:41:45.849234 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 8.402053ms to process 0 records I0320 12:41:45.849250 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 7.586452ms to process 0 records I0320 12:41:45.849257 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 8.110752ms to process 0 records I0320 12:41:45.849263 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 8.3148ms to process 0 records I0320 12:41:45.849271 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 7.772818ms to process 0 records I0320 12:41:45.849281 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 8.530552ms to process 0 records I0320 12:41:45.849290 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 8.558417ms to process 0 records I0320 12:41:45.849298 1 tasks_processing.go:71] worker 21 working on olm_operators task. E0320 12:41:45.849414 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0320 12:41:45.849433 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 8.541349ms to process 0 records I0320 12:41:45.849447 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 8.462423ms to process 0 records I0320 12:41:45.849519 1 tasks_processing.go:71] worker 26 working on nodenetworkstates task. I0320 12:41:45.849568 1 tasks_processing.go:71] worker 7 working on active_alerts task. I0320 12:41:45.849527 1 tasks_processing.go:71] worker 24 working on version task. I0320 12:41:45.849540 1 tasks_processing.go:71] worker 31 working on openshift_logging task. E0320 12:41:45.850231 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.33.180:9091: connect: connection refused I0320 12:41:45.850356 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0320 12:41:45.850379 1 controller.go:368] The initial operator extension status is healthy I0320 12:41:45.850390 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0320 12:41:45.850396 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0320 12:41:45.850401 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0320 12:41:45.849545 1 tasks_processing.go:71] worker 10 working on image_pruners task. I0320 12:41:45.850443 1 controller.go:531] The operator is still being initialized I0320 12:41:45.850453 1 controller.go:554] The operator is healthy I0320 12:41:45.850710 1 recorder.go:75] Recording config/infrastructure with fingerprint=dc36347d8e96217e81fdbe06ece33e993afba4cae4bf1fa68c0b7755dce1766b I0320 12:41:45.850764 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0320 12:41:45.850776 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 8.763145ms to process 1 records I0320 12:41:45.850793 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 8.522278ms to process 0 records I0320 12:41:45.850903 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=957e1947baac75e4f0ffee28af05dd2595f7f370b66dd4fd5aa357214b22ce66 I0320 12:41:45.850944 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 9.700688ms to process 1 records I0320 12:41:45.851132 1 tasks_processing.go:71] worker 28 working on nodes task. I0320 12:41:45.851200 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0320 12:41:45.851261 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0320 12:41:45.850735 1 operator.go:328] started I0320 12:41:45.849556 1 tasks_processing.go:71] worker 17 working on image task. I0320 12:41:45.851464 1 tasks_processing.go:71] worker 13 working on storage_cluster task. I0320 12:41:45.849561 1 tasks_processing.go:71] worker 5 working on certificate_signing_requests task. E0320 12:41:45.849560 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0320 12:41:45.849572 1 tasks_processing.go:71] worker 14 working on machine_healthchecks task. I0320 12:41:45.849548 1 tasks_processing.go:71] worker 16 working on cost_management_metrics_configs task. I0320 12:41:45.851677 1 recorder.go:75] Recording config/oauth with fingerprint=4244bdb3f23edc920d50ea1b62d9676a3ef53d46fd01e75328646249b6c817e6 I0320 12:41:45.851921 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 9.546612ms to process 1 records I0320 12:41:45.851676 1 tasks_processing.go:71] worker 3 working on machine_configs task. I0320 12:41:45.851977 1 tasks_processing.go:71] worker 20 working on openshift_machine_api_events task. E0320 12:41:45.852593 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.33.180:9091: connect: connection refused E0320 12:41:45.851964 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.33.180:9091: connect: connection refused I0320 12:41:45.852805 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 9.21256ms to process 0 records E0320 12:41:45.852820 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0320 12:41:45.852829 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 10.275711ms to process 0 records I0320 12:41:45.852836 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 7.575093ms to process 0 records I0320 12:41:45.852842 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 11.350255ms to process 0 records E0320 12:41:45.852849 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.33.180:9091: connect: connection refused I0320 12:41:45.852910 1 tasks_processing.go:71] worker 2 working on ingress task. I0320 12:41:45.852918 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 12.292585ms to process 0 records I0320 12:41:45.852880 1 tasks_processing.go:71] worker 1 working on image_registries task. I0320 12:41:45.852896 1 tasks_processing.go:71] worker 23 working on mutating_webhook_configurations task. I0320 12:41:45.852904 1 tasks_processing.go:71] worker 19 working on aggregated_monitoring_cr_names task. I0320 12:41:45.854586 1 tasks_processing.go:71] worker 6 working on proxies task. E0320 12:41:45.854802 1 gather_active_alerts.go:64] Unable to retrieve most recent alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp 172.30.26.239:9094: connect: connection refused I0320 12:41:45.854832 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=270cdd6ec66022d8edae1e195e5f5f16a96aefebb882f32e46beb6baf1828d89 I0320 12:41:45.854955 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=97a3f5dfa484d68a4ece18f9e15701a28a7a1ef6cce85048ef9e0c571e5f52e1 I0320 12:41:45.854972 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 13.105624ms to process 2 records E0320 12:41:45.854981 1 gather.go:140] gatherer "clusterconfig" function "active_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp 172.30.26.239:9094: connect: connection refused I0320 12:41:45.854995 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 5.290056ms to process 0 records I0320 12:41:45.855013 1 tasks_processing.go:71] worker 7 working on sap_config task. I0320 12:41:45.855218 1 tasks_processing.go:71] worker 25 working on silenced_alerts task. I0320 12:41:45.855332 1 recorder.go:75] Recording config/authentication with fingerprint=89f63704fc8e3f077c5930401c95b9e9c8f5ab3e4cd7ab9b25b69e0da020ed9d I0320 12:41:45.855385 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 7.0798ms to process 1 records I0320 12:41:45.855488 1 tasks_processing.go:71] worker 12 working on machine_sets task. I0320 12:41:45.855585 1 recorder.go:75] Recording config/network with fingerprint=d6aa9376a16afa15812954326591985217bac7e43c4752e25aacda218dfeb618 I0320 12:41:45.855603 1 gather.go:177] gatherer "clusterconfig" function "networks" took 10.864608ms to process 1 records W0320 12:41:45.857253 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. E0320 12:41:45.857607 1 gather_silenced_alerts.go:52] Unable to retrieve silenced alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp 172.30.26.239:9094: connect: connection refused I0320 12:41:45.857625 1 tasks_processing.go:71] worker 25 working on overlapping_namespace_uids task. E0320 12:41:45.857674 1 gather.go:140] gatherer "clusterconfig" function "silenced_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp 172.30.26.239:9094: connect: connection refused I0320 12:41:45.857696 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 2.395833ms to process 0 records I0320 12:41:45.864292 1 tasks_processing.go:71] worker 13 working on monitoring_persistent_volumes task. I0320 12:41:45.864303 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 12.805296ms to process 0 records I0320 12:41:45.864450 1 tasks_processing.go:71] worker 14 working on pdbs task. E0320 12:41:45.864475 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0320 12:41:45.864488 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 12.707144ms to process 0 records I0320 12:41:45.864514 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 19.500478ms to process 0 records I0320 12:41:45.864580 1 tasks_processing.go:71] worker 8 working on qemu_kubevirt_launcher_logs task. I0320 12:41:45.870274 1 tasks_processing.go:71] worker 30 working on feature_gates task. I0320 12:41:45.870284 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 28.613028ms to process 0 records I0320 12:41:45.870361 1 tasks_processing.go:71] worker 31 working on clusterroles task. I0320 12:41:45.870376 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 20.163935ms to process 0 records I0320 12:41:45.870574 1 tasks_processing.go:71] worker 17 working on lokistack task. I0320 12:41:45.870770 1 recorder.go:75] Recording config/image with fingerprint=0f12dfb21c78d7efb3a0079ca74c455781fa5948367e2f332af02bf56b6370e7 I0320 12:41:45.870784 1 gather.go:177] gatherer "clusterconfig" function "image" took 19.126949ms to process 1 records I0320 12:41:45.870790 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 18.790647ms to process 0 records E0320 12:41:45.870796 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0320 12:41:45.870802 1 tasks_processing.go:71] worker 16 working on ingress_certificates task. I0320 12:41:45.870803 1 gather.go:177] gatherer "clusterconfig" function "machines" took 25.774501ms to process 0 records I0320 12:41:45.870836 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 21.05347ms to process 0 records I0320 12:41:45.870847 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 18.686298ms to process 0 records I0320 12:41:45.870857 1 tasks_processing.go:71] worker 20 working on service_accounts task. I0320 12:41:45.870810 1 tasks_processing.go:71] worker 0 working on operators task. I0320 12:41:45.870973 1 tasks_processing.go:71] worker 26 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0320 12:41:45.871620 1 tasks_processing.go:74] worker 12 stopped. I0320 12:41:45.871636 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 16.074967ms to process 0 records I0320 12:41:45.871696 1 tasks_processing.go:74] worker 10 stopped. I0320 12:41:45.872092 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=83776731085341481d6f98e2ae1abd756ff72ef133e5331350db0cf2cc8d1a03 I0320 12:41:45.872174 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 21.283394ms to process 1 records I0320 12:41:45.872706 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=f88c6fdf70d7520ed886cf6dda35bc5a14c6ebd2287066ce9dd7f7b66f79bd30 I0320 12:41:45.872762 1 tasks_processing.go:74] worker 18 stopped. I0320 12:41:45.872907 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=ab93ccc83fff79a352bbeb5756f2297ede2f265e3f2187126ff5dbe77ad4ed09 I0320 12:41:45.873196 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=4f081647adf129e932b82f3a806a096df7911944d9c1e2e88f756894ea097e91 I0320 12:41:45.873300 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=652ca180d7e21e095173588fafc0e1ade5eb0b7fada8d192d4d68b5c2dc13adf I0320 12:41:45.873368 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=87f202db3593557723a026b56599a7b0acc5fb3bd7614f653a683ec675e709ed I0320 12:41:45.873419 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=f6fef4a96ea7f67b666fb417542c86406663b0f8075b2bec897a4a1426d6d5c2 I0320 12:41:45.873489 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=3f06c081787968673ad9526f0398a3cc94b90b81f0c91561c90cc0acbfc11c9b I0320 12:41:45.873553 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=4928aa1aa34e7714c2a934d14a2d69509975f1cf2f439b8c944610c6d7d0c3d8 I0320 12:41:45.873610 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=834a05bd2b38cfd128b7b3e50fee17f9dce1abc42274179901b252915a02b113 I0320 12:41:45.873697 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=12ac2dd7da056ca10553d1be7acc090d85a6275c9144b7596954778c40e169f5 I0320 12:41:45.873754 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=6a2fd0af5e0c3742b516e1581412bb2d0f4df498f9215f30d87a488d59d92215 I0320 12:41:45.873814 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=c63f97d402c6d25bf6f2e047a5a32ab1b9a3c9f79dc33a78bae43e5ea7bc4021 I0320 12:41:45.873866 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=7a98923a8839169b8b9b553959fd562cf8a02355126b073c253088c480fc2fbe I0320 12:41:45.873900 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 30.746465ms to process 13 records I0320 12:41:45.874072 1 tasks_processing.go:74] worker 2 stopped. I0320 12:41:45.874237 1 recorder.go:75] Recording config/ingress with fingerprint=8428187bc4ac3a6449d2b817bbbbf0063974eba1c94f5e12568c7aeef9d5b6ae I0320 12:41:45.874286 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 19.760909ms to process 1 records I0320 12:41:45.874317 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 17.771186ms to process 0 records I0320 12:41:45.874477 1 tasks_processing.go:74] worker 7 stopped. I0320 12:41:45.874543 1 tasks_processing.go:74] worker 9 stopped. I0320 12:41:45.875624 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=369ee133f54ede17587e7e06afae1833ad6f1641ac4fa9ce21e67eef91f33f96 I0320 12:41:45.876279 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=0da6e614b7b10d9fb108c111ada56a4eed332fbbcd6c6a5c340766529faec59e I0320 12:41:45.876299 1 gather.go:177] gatherer "clusterconfig" function "crds" took 31.582111ms to process 2 records I0320 12:41:45.876429 1 tasks_processing.go:74] worker 1 stopped. I0320 12:41:45.876707 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=f277342b6009d3c3b690cc94e35a4fc45f57171933603b96797445ad322e7046 I0320 12:41:45.876723 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 20.687046ms to process 1 records I0320 12:41:45.876812 1 tasks_processing.go:74] worker 28 stopped. I0320 12:41:45.876985 1 recorder.go:75] Recording config/node/ip-10-0-0-209.ec2.internal with fingerprint=ba349d52c426c87608fdd4d361b12e40620cf64b1b4643fea8d18142a8c74b4c I0320 12:41:45.877046 1 recorder.go:75] Recording config/node/ip-10-0-1-246.ec2.internal with fingerprint=271c1db1d7f4c703a5459da465f074d9839c025033f05dfbe8bbec4546fdd4f8 I0320 12:41:45.877117 1 recorder.go:75] Recording config/node/ip-10-0-2-155.ec2.internal with fingerprint=d3d1ee44d92b650902d6cf70dffb5d5cfaaed60d28a57672ea181463e74c61b0 I0320 12:41:45.877131 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 22.820468ms to process 3 records E0320 12:41:45.877143 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0320 12:41:45.877174 1 tasks_processing.go:74] worker 3 stopped. I0320 12:41:45.877181 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0320 12:41:45.877192 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 23.487386ms to process 1 records I0320 12:41:45.877260 1 recorder.go:75] Recording config/proxy with fingerprint=8aa73932790004e3578b349402ebeee5733c75e4d038ddb7d7ab7b24f54eb8bd I0320 12:41:45.877274 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 21.523897ms to process 1 records I0320 12:41:45.877403 1 recorder.go:75] Recording config/apiserver with fingerprint=82467fb4657fab7993424cc82b1d6b16abeb58cbea5b0f75d2bac01768b83722 I0320 12:41:45.877415 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 27.831464ms to process 1 records I0320 12:41:45.877433 1 tasks_processing.go:74] worker 6 stopped. I0320 12:41:45.877462 1 tasks_processing.go:74] worker 4 stopped. I0320 12:41:45.877535 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=c310a66d1ae5c99aa3129e6dae8621377302e3610889f7dd7b62dbc94ade7d61 I0320 12:41:45.877548 1 tasks_processing.go:74] worker 23 stopped. I0320 12:41:45.877583 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=bee47ec50784536f26e6d53e159a023ec19296a84cf1697bde637eb57a65f32a I0320 12:41:45.877661 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=efa8826c5022e9d377da49109bd5f9d392bd47730feace43ad694e4fc0d56a79 I0320 12:41:45.877693 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 23.549287ms to process 3 records I0320 12:41:45.877712 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0320 12:41:45.877721 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 19.745264ms to process 1 records I0320 12:41:45.877724 1 tasks_processing.go:74] worker 25 stopped. I0320 12:41:45.877729 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 6.875649ms to process 0 records I0320 12:41:45.877734 1 tasks_processing.go:74] worker 17 stopped. I0320 12:41:45.878992 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0320 12:41:45.879006 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0320 12:41:45.879011 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0320 12:41:45.879016 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0320 12:41:45.879021 1 controller.go:254] Source scaController *sca.Controller is not ready I0320 12:41:45.879073 1 controller.go:531] The operator is still being initialized I0320 12:41:45.879085 1 controller.go:554] The operator is healthy I0320 12:41:45.879520 1 tasks_processing.go:74] worker 5 stopped. I0320 12:41:45.879539 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 27.850295ms to process 0 records I0320 12:41:45.888763 1 tasks_processing.go:74] worker 24 stopped. I0320 12:41:45.888959 1 gather_logs.go:145] no pods in namespace were found I0320 12:41:45.889089 1 recorder.go:75] Recording config/version with fingerprint=a400463c1d94e75b64f7b976fd79418cbcf322a4f4f6f7190ddd027200ab6e24 I0320 12:41:45.889104 1 recorder.go:75] Recording config/id with fingerprint=0c109ba90bf073c3b7203c1cf9ff91ebfb30339eb3997e2741411daf06af957a I0320 12:41:45.889113 1 gather.go:177] gatherer "clusterconfig" function "version" took 39.05967ms to process 2 records I0320 12:41:45.889199 1 tasks_processing.go:74] worker 30 stopped. I0320 12:41:45.889249 1 recorder.go:75] Recording config/featuregate with fingerprint=bbca107613f1196ea262fbcf794c704b465272c82efa2c2abf4bded6d8847852 I0320 12:41:45.889260 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 18.633759ms to process 1 records I0320 12:41:45.889271 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 24.382813ms to process 0 records I0320 12:41:45.889279 1 tasks_processing.go:74] worker 8 stopped. I0320 12:41:45.890414 1 tasks_processing.go:74] worker 21 stopped. I0320 12:41:45.890500 1 recorder.go:75] Recording config/olm_operators with fingerprint=140b062a856e44c0917a425514fa831f64958088bedb497702072125ee4bc357 I0320 12:41:45.890516 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 41.105435ms to process 1 records I0320 12:41:45.893593 1 tasks_processing.go:74] worker 13 stopped. I0320 12:41:45.893638 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 29.283363ms to process 0 records I0320 12:41:45.894145 1 tasks_processing.go:74] worker 14 stopped. I0320 12:41:45.894322 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=56065222f240820b5361cd084cd1f73b24c0281ad9e9d105f7199a3b96d9e0ea I0320 12:41:45.894351 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=03eef8aef87b90c839a90af49e7b8d18cd83eed084223776560bc518246842e2 I0320 12:41:45.894370 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=279a8f862cc81653345d9c8ae064620a00df817b4e88ff278c4da7122466ee23 I0320 12:41:45.894387 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=0a701595c4a191628e05be059243fb7d08e485068e8c505a22b85e6cc44624b9 I0320 12:41:45.894409 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=c50a8a20e5934be0903d29d0896f150e4b7077415eb9557ca9ccedf9f1a03e18 I0320 12:41:45.894432 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=3d6cffb6c97aa181c61c045f58ec5addbbf09636744e7a194c9ad9a3130f7fcd I0320 12:41:45.894452 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=12a727433a1b2b87cabda003d73bc65491dadfc36ca1f0f13a245b39f9f3d376 I0320 12:41:45.894474 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=513285f5312e5c1038050e272003aa0112ddac2e95abff5ba8fc87a0cbd5586e I0320 12:41:45.894496 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=747c4dc497f1e33efc622535e0c66fe45f15516d7ecf16e88352c44dbf0cac1a I0320 12:41:45.894521 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=e4459c1902091eeff321cc5a6b51e78e1175e8795282c65204335e58d5d62770 I0320 12:41:45.894567 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 29.679851ms to process 10 records I0320 12:41:45.895998 1 tasks_processing.go:74] worker 31 stopped. I0320 12:41:45.896487 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=dc11920a084a98191a524b252f7ac3c62d615f01f1ba109181a1c2962c7ab3bd I0320 12:41:45.896724 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=c0e867b9bc0568a5d1d2a27d8d45108e1c1caf0fedc8597a5ecb2d4e55b6cf0a I0320 12:41:45.896772 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 25.606607ms to process 2 records I0320 12:41:45.896885 1 tasks_processing.go:74] worker 19 stopped. I0320 12:41:45.896919 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 43.680209ms to process 0 records I0320 12:41:45.901988 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0320 12:41:45.903417 1 tasks_processing.go:74] worker 27 stopped. I0320 12:41:45.905230 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-5b5d9b4544-686cs with fingerprint=340076d981174fc0ce13d69f9b53b32b7befa821978cbc479b8470ae846a6cee I0320 12:41:45.905488 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-w7mvs with fingerprint=21ee55883469dfb5cd5fa2e67ca4c5e163fbd5d4448396cafd8a6fb6ebab29e7 I0320 12:41:45.905717 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-xw69w with fingerprint=0a9466a36db791936239d365b1d4903173d058881503421668fb3ca30022cb12 I0320 12:41:45.905785 1 recorder.go:75] Recording config/running_containers with fingerprint=670132fe4d41ae95ef9a04ec8163b07d7d8c6a8aa43dfc6ec7cf769ec4ba3f63 I0320 12:41:45.905795 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 61.711958ms to process 4 records I0320 12:41:45.907975 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0320 12:41:45.908010 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0320 12:41:45.908096 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0320 12:41:45.912142 1 tasks_processing.go:74] worker 29 stopped. E0320 12:41:45.912209 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0320 12:41:45.912258 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0320 12:41:45.912287 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0320 12:41:45.912320 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=c8a9805405aae7d04b7f03c09c51097942070877009b6daebd5573ef8127446e I0320 12:41:45.912424 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0320 12:41:45.912463 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0320 12:41:45.912490 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=3b70b9787487b548204d806ebebf90b0c2d460145e34cb33c0d8d3a7a011d907 I0320 12:41:45.912526 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0320 12:41:45.912607 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0320 12:41:45.912656 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0320 12:41:45.912686 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 70.777562ms to process 7 records I0320 12:41:45.917168 1 base_controller.go:82] Caches are synced for ConfigController I0320 12:41:45.917183 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0320 12:41:45.919230 1 tasks_processing.go:74] worker 26 stopped. I0320 12:41:45.919250 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 48.205519ms to process 0 records I0320 12:41:45.922081 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 12:41:45.951312 1 base_controller.go:82] Caches are synced for LoggingSyncer I0320 12:41:45.951324 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0320 12:41:45.952923 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0320 12:41:45.953750 1 cluster_transfer.go:102] no available accepted cluster transfer I0320 12:41:45.953763 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0320 12:41:45.956421 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-86967b9448-fcfbs pod in namespace openshift-ingress (previous: false). I0320 12:41:45.986264 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-bjlhf pod in namespace openshift-ingress-canary (previous: false). I0320 12:41:45.996636 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-f5pd9 pod in namespace openshift-ingress-canary (previous: false). I0320 12:41:46.006704 1 tasks_processing.go:74] worker 16 stopped. E0320 12:41:46.006716 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0320 12:41:46.006722 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2p5etjbdo3k0bg4iueo4ra70s5l4l6pc-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2p5etjbdo3k0bg4iueo4ra70s5l4l6pc-primary-cert-bundle-secret" not found I0320 12:41:46.006786 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=47fa5f270c202650841f9f68dc5ad3063f08cf993d9c1b2c8325362e11549323 I0320 12:41:46.006799 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 135.880495ms to process 1 records I0320 12:41:46.050865 1 sca.go:266] etc-pki-entitlement secret successfully created I0320 12:41:46.050884 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0320 12:41:46.072187 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-spqwv pod in namespace openshift-ingress-canary (previous: false). I0320 12:41:46.273112 1 tasks_processing.go:74] worker 15 stopped. I0320 12:41:46.273207 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=1e2ef370724c24753fdec60b66aa1173f2f61673a61115175cd78f0771b22a1a I0320 12:41:46.273296 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=99fe20ce13ce5c3f9f38e7f24fb1f879ae74452f92228ece69d4552deb1358d3 I0320 12:41:46.273335 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=b453273e7cd944efcf7207d8882805330194d206628a8a16bb55c9acc90233c6 I0320 12:41:46.273374 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-5774cb4f57-7gvb5/router_current.log with fingerprint=e5922c5cead67a6e05ae987a23f045d24c1cd0fb55dcaef357bbe85a106153d2 I0320 12:41:46.273410 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-86967b9448-9d4vz/router_current.log with fingerprint=4427613568afe1cdc44117ee715cf53ad8fe18759dbcc93bca51c4a6c5f556b7 I0320 12:41:46.273454 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-86967b9448-fcfbs/router_current.log with fingerprint=49486acb3d5e98aacbafe5dafd443509ff23004abc7d550da154861b50218d04 I0320 12:41:46.273461 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-bjlhf/serve-healthcheck-canary_current.log with fingerprint=690b69a18baf10a0968722dd877a040952312426d2d671864b19478f09e9f83c I0320 12:41:46.273465 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-f5pd9/serve-healthcheck-canary_current.log with fingerprint=d8346eecdd3b87ee20c6a9b5e69bdd3fe62a0dd5f19c8796c8158fef45653750 I0320 12:41:46.273469 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-spqwv/serve-healthcheck-canary_current.log with fingerprint=15f490eb35c60bebb8089c99de4d4d378712c8c77f636a722da17b4363b58b46 I0320 12:41:46.273476 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 432.151762ms to process 9 records W0320 12:41:46.856994 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0320 12:41:47.857323 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 12:41:48.313340 1 tasks_processing.go:74] worker 0 stopped. I0320 12:41:48.313473 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=b7f5e825557e084925af165ac5485498fb2825d172599f6458fe48d0bec50e0d I0320 12:41:48.313539 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0320 12:41:48.313587 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=886aba238e52cd1b17be701daf3ea790d92e518962183748d7f2a98d95e58c0f I0320 12:41:48.313602 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0320 12:41:48.313634 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=341df6e915e6e78852a89b76ef6c6ff0669ec917e9cc9f3703404dd25fe5cb55 I0320 12:41:48.313659 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0320 12:41:48.313696 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=105a5a1ca9c79ba7da1fb832a54196bef892eb5dcad156ee49af9af87c742115 I0320 12:41:48.313736 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=160cf7c30022c7fd1d4f73c54d6a980b868e21bfbc199a6b78f10f4d473bc8ee I0320 12:41:48.313771 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=6562e339c683a741610a36006abf5b483a04178d7b8b1a77a594eed393dd01d9 I0320 12:41:48.313815 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=b694f8d988bb3bd0f89cec2b5a37c9065960dd23704653f30fd3acb7e55ad338 I0320 12:41:48.314103 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=6824024073cf344934a934a780f20f08d70ed63d43426484fe29f6383603f089 I0320 12:41:48.314138 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=1a6234a194078e9a0d2a140ec35f832f962850ff80b68985b92e2aa92ff6060a I0320 12:41:48.314151 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0320 12:41:48.314177 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=d550df24991d35ae47ff1f8221dea9eaf3bc52c3962f012d5368ccf35db58705 I0320 12:41:48.314193 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0320 12:41:48.314220 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=fa412b25dba811f88271170c89ceaa4b4c0806414dce200508325a39e336af28 I0320 12:41:48.314234 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0320 12:41:48.314265 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=964d3dff01cc8611a39e0501a94885fc43226b6feb3d2aa9931f9d98907717d1 I0320 12:41:48.314279 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0320 12:41:48.314311 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=8b4ef8f9a6f8dc6f0db357d67b861c14605d94aacb0022bdd4f6f9501f6508a5 I0320 12:41:48.314427 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=89cad34cd8566896c70ac4fd54358750f2377259cd196471c0d5dec56d2fee66 I0320 12:41:48.314441 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0320 12:41:48.314454 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0320 12:41:48.314487 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0320 12:41:48.314523 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=c14d052736329cfcb0ca26e971ffff57405a180b1b0b972e0fc1c4a5ab2cff6b I0320 12:41:48.314560 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=883660e4be4fe90d0a36d8e7e902ffa53daf284d7f15d42622bfd9cbb76ea18a I0320 12:41:48.314574 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0320 12:41:48.314603 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=76fc85f966e811f08c56efc5f583f14edc8cb59547af093602222ef364970435 I0320 12:41:48.314618 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0320 12:41:48.314646 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=47d12721ffaa81a6361eefaea585dc156b643c167cc8c125f16fc2f2a412651d I0320 12:41:48.314660 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0320 12:41:48.314684 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=d8b87fc56b345d9fb2447821b6a1d90fb75f1737a6ad6d65636e29819d26aa04 I0320 12:41:48.314712 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=af9dcb68f8ff32b79253fc049038cea732b64735e1f0d48280e9258ec13a9082 I0320 12:41:48.314737 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=f22b5dbf744a12f08d23219414ad79820ff8697ac7a07690c298fdbb8119e0c8 I0320 12:41:48.314774 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=e142f061d3cf5d18f6a6cefdf0acc099e71f37e70453a2e467910fe4ee98ae69 I0320 12:41:48.314788 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0320 12:41:48.314824 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=ec838845c3f5b3858e57cee976fd2ae79ce9d1ccce037d2a52d3bf07bb82013d I0320 12:41:48.314851 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0320 12:41:48.314862 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0320 12:41:48.314875 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.442400402s to process 39 records W0320 12:41:48.857428 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 12:41:49.862208 1 tasks_processing.go:74] worker 11 stopped. I0320 12:41:49.862231 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=108780cb24f15bf572a4031966507398a0d0c6a28b3a008952f53b481dc191a9 I0320 12:41:49.862277 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 4.018377603s to process 1 records I0320 12:41:58.670784 1 tasks_processing.go:74] worker 22 stopped. I0320 12:41:58.670829 1 recorder.go:75] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0320 12:41:58.670843 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.829205982s to process 1 records I0320 12:41:59.476574 1 tasks_processing.go:74] worker 20 stopped. I0320 12:41:59.476845 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=168fffbb2ffe3e4223178b9929987b11ee2f64fe6f0ebe7873bbab778d18dbe6 I0320 12:41:59.476862 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.605700798s to process 1 records E0320 12:41:59.476935 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.636s with: function \"pod_network_connectivity_checks\" failed with an error, function \"tsdb_status\" failed with an error, function \"support_secret\" failed with an error, function \"metrics\" failed with an error, function \"active_alerts\" failed with an error, function \"silenced_alerts\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error" I0320 12:41:59.478042 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "tsdb_status" failed with an error, function "support_secret" failed with an error, function "metrics" failed with an error, function "active_alerts" failed with an error, function "silenced_alerts" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error I0320 12:41:59.478073 1 periodic.go:216] Running workloads gatherer I0320 12:41:59.478090 1 tasks_processing.go:45] number of workers: 2 I0320 12:41:59.478095 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 12:41:59.478099 1 tasks_processing.go:71] worker 1 working on helmchart_info task. I0320 12:41:59.478116 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 12:41:59.478187 1 tasks_processing.go:71] worker 0 working on workload_info task. I0320 12:41:59.483727 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-209.ec2.internal... I0320 12:41:59.483737 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-246.ec2.internal... I0320 12:41:59.483756 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-155.ec2.internal... I0320 12:41:59.497820 1 tasks_processing.go:74] worker 1 stopped. I0320 12:41:59.497837 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 19.705795ms to process 0 records I0320 12:42:04.290333 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 5s I0320 12:42:04.315542 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 25s for image data I0320 12:42:04.324342 1 gather_workloads_info.go:387] No image sha256:1aabd00f813e1afccdab16a845e6346288f5cf7e320fa5bb97c683702191c575 (10ms) I0320 12:42:04.332669 1 gather_workloads_info.go:387] No image sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319 (8ms) I0320 12:42:04.341331 1 gather_workloads_info.go:387] No image sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6 (9ms) I0320 12:42:04.349733 1 gather_workloads_info.go:387] No image sha256:7db22f56cf9d223b1437e8cba3aa9e04e8ae8b973e23e85133bf1d3d8d66f15e (8ms) I0320 12:42:04.359357 1 gather_workloads_info.go:387] No image sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f (10ms) I0320 12:42:04.367347 1 gather_workloads_info.go:387] No image sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044 (8ms) I0320 12:42:04.374877 1 gather_workloads_info.go:387] No image sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648 (8ms) I0320 12:42:04.383099 1 gather_workloads_info.go:387] No image sha256:2af652f68b17ed49c2565928e4c101880e949c4484c6ee958793a8e668e47bea (8ms) I0320 12:42:04.390693 1 gather_workloads_info.go:387] No image sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211 (8ms) I0320 12:42:04.398503 1 gather_workloads_info.go:387] No image sha256:876fb57ffb255ed75a452405bcd1d116788a34b19264886e6e087ff544c17083 (8ms) I0320 12:42:04.423082 1 gather_workloads_info.go:387] No image sha256:a9dc28a6962d195dcce0c76a544aad5f784dd90b98282c169d7b94c870a85f8b (25ms) I0320 12:42:04.524853 1 gather_workloads_info.go:387] No image sha256:2962c6c31d7078b34cf7ba0c9b0ea685dc539cea67b06bd38b6ac566298bb790 (102ms) I0320 12:42:04.625488 1 gather_workloads_info.go:387] No image sha256:07f14370b57046dc2ac49be933bb2248aaf5943e8fb56ce9703eaa7c673b999f (101ms) I0320 12:42:04.724617 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (99ms) I0320 12:42:04.825147 1 gather_workloads_info.go:387] No image sha256:3e772c41ef85aaccf63c409ee72b43d632b36d4cc66a3790835dffdcf9b7433e (101ms) I0320 12:42:04.925297 1 gather_workloads_info.go:387] No image sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498 (100ms) I0320 12:42:05.024132 1 gather_workloads_info.go:387] No image sha256:9dee9b6866db30456c277ff1f7a114dd4ce25bff8aced70218623af9423ec3cf (99ms) I0320 12:42:05.125268 1 gather_workloads_info.go:387] No image sha256:0b2cc6aadb1abc1d35a55da0c9884eb9426d7ff7a97174daeb09bb1e1a23cfae (101ms) I0320 12:42:05.227572 1 gather_workloads_info.go:387] No image sha256:03a7d45ceb838c746bdcdac1a5cc02d26273b319d9da4498bb3ee31c3e130b43 (102ms) I0320 12:42:05.327613 1 gather_workloads_info.go:387] No image sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd (100ms) I0320 12:42:05.424885 1 gather_workloads_info.go:398] Found image sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd (97ms) I0320 12:42:05.525005 1 gather_workloads_info.go:387] No image sha256:50b7c5185c4a6e63145753415c92723c59136d14b951ea0f4f06739b00893329 (100ms) I0320 12:42:05.624631 1 gather_workloads_info.go:387] No image sha256:ce98d5d844bfc2ba8de1893866ad38166c95157d54abd8192b181e819bc50bb5 (100ms) I0320 12:42:05.724522 1 gather_workloads_info.go:387] No image sha256:188e4e2d697e2ca3c7c16772c0ab12253f00bd8433d8691ed7058fe7bafbbbf4 (100ms) I0320 12:42:05.824982 1 gather_workloads_info.go:387] No image sha256:3e889ea85c9c4748c548b84ae135a8adc88f42026ce8073538ca2fe7d89b8004 (100ms) I0320 12:42:05.925565 1 gather_workloads_info.go:387] No image sha256:11de6fe1741f5dd485e59ee2e703868badda81de0a179513be70b6ac9fc4655f (101ms) I0320 12:42:06.024634 1 gather_workloads_info.go:387] No image sha256:7f426776d3ef70b585018b8d813f5eedd70373536681c6240df224ebf9ff6329 (99ms) I0320 12:42:06.124531 1 gather_workloads_info.go:387] No image sha256:8f13a1e673c882ee926e97270249a20d3c40797a8c7c5abb2fda9c98c24a2eb4 (100ms) I0320 12:42:06.224232 1 gather_workloads_info.go:387] No image sha256:30a054cfe1b8330bd57d762921ca5cd281ad2fdd7994fd3093ffadb5d767a564 (100ms) I0320 12:42:06.325044 1 gather_workloads_info.go:387] No image sha256:3666bcb850e1518aec86d322af52e95f725602ad0ab770d2b1e5b39b6dbde0b5 (101ms) I0320 12:42:06.424963 1 gather_workloads_info.go:387] No image sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e (100ms) I0320 12:42:06.525116 1 gather_workloads_info.go:387] No image sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc (100ms) I0320 12:42:06.625284 1 gather_workloads_info.go:387] No image sha256:6521242b8805e70936f8e49c7d50a319adaeb73b4e5dec4693b0f93a34e397c5 (100ms) I0320 12:42:06.723840 1 gather_workloads_info.go:387] No image sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c (99ms) I0320 12:42:06.824775 1 gather_workloads_info.go:387] No image sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809 (101ms) I0320 12:42:06.924343 1 gather_workloads_info.go:387] No image sha256:1ff360b712000d39cbb58158186ff0a18764278e6fa98ef9f21d113b978d9a79 (100ms) I0320 12:42:07.026222 1 gather_workloads_info.go:387] No image sha256:3d68a9523f4c5c22a48dfa151bbfde78b2f360697afe3b9d8bfb53e8c359bf95 (102ms) I0320 12:42:07.127300 1 gather_workloads_info.go:387] No image sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df (101ms) I0320 12:42:07.224719 1 gather_workloads_info.go:387] No image sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958 (97ms) I0320 12:42:07.325003 1 gather_workloads_info.go:387] No image sha256:12e6985890d3c549f7cc9881892793b27277f72e2739cb910aaf9f4dd7215df1 (100ms) I0320 12:42:07.424939 1 gather_workloads_info.go:387] No image sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb (100ms) I0320 12:42:07.524995 1 gather_workloads_info.go:387] No image sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635 (100ms) I0320 12:42:07.624649 1 gather_workloads_info.go:387] No image sha256:317131706dcce9c87eb4e1efb8446bdbf5e0726e120a8f6c655e2acd89d60e2d (100ms) I0320 12:42:07.724584 1 gather_workloads_info.go:387] No image sha256:60648d27017709a420e43353f9a83d7c466017b3b771200672a4dbc3c3f7a3ae (100ms) I0320 12:42:07.824609 1 gather_workloads_info.go:387] No image sha256:124f1b391b0f376a5deffd05b29ef0896ca0912e1f67b3a2051ed6678c54a57e (100ms) I0320 12:42:07.925170 1 gather_workloads_info.go:387] No image sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6 (101ms) I0320 12:42:08.024277 1 gather_workloads_info.go:398] Found image sha256:670004bffddb41c787b9f9a5c7d1615453d4acfa6452dedfa61ad2f305eea814 (99ms) I0320 12:42:08.124178 1 gather_workloads_info.go:387] No image sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7 (100ms) I0320 12:42:08.124216 1 tasks_processing.go:74] worker 0 stopped. I0320 12:42:08.124590 1 recorder.go:75] Recording config/workload_info with fingerprint=1dd861c752af9c84dae27478ea87ad7b182f4c1a0f639c5f6b314fec779e3d4d I0320 12:42:08.124612 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.646016171s to process 1 records I0320 12:42:08.124647 1 periodic.go:263] Periodic gather workloads completed in 8.646s I0320 12:42:08.124661 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0320 12:42:08.124668 1 periodic.go:216] Running conditional gatherer I0320 12:42:08.129134 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.5/gathering_rules I0320 12:42:08.229030 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer E0320 12:42:08.231206 1 conditional_gatherer.go:324] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp 172.30.33.180:9091: connect: connection refused I0320 12:42:08.231280 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0320 12:42:08.236457 1 conditional_gatherer.go:392] cluster version is '4.21.5' E0320 12:42:08.236469 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:42:08.236474 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:42:08.236478 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:42:08.236482 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:42:08.236485 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:42:08.236488 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:42:08.236490 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:42:08.236493 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:42:08.236496 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:42:08.236499 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0320 12:42:08.236513 1 tasks_processing.go:45] number of workers: 3 I0320 12:42:08.236529 1 tasks_processing.go:69] worker 2 listening for tasks. I0320 12:42:08.236535 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0320 12:42:08.236540 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 12:42:08.236549 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0320 12:42:08.236555 1 tasks_processing.go:71] worker 0 working on conditional_gatherer_rules task. I0320 12:42:08.236555 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 12:42:08.236567 1 tasks_processing.go:74] worker 1 stopped. I0320 12:42:08.236635 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0320 12:42:08.236655 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.164µs to process 1 records I0320 12:42:08.236729 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=9fafdcbf373f1f367f1d69551ea59114592cce1681189be0f3dde774e0530661 I0320 12:42:08.236741 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 684ns to process 1 records I0320 12:42:08.236744 1 tasks_processing.go:74] worker 0 stopped. I0320 12:42:08.236768 1 tasks_processing.go:74] worker 2 stopped. I0320 12:42:08.236779 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 227.699µs to process 0 records I0320 12:42:08.236800 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0320 12:42:08.236817 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0320 12:42:08.259466 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=689440929a68da32b5d923202a5ca55a002bd194351d050a88e41d9d5daf85e7 I0320 12:42:08.259589 1 diskrecorder.go:70] Writing 119 records to /var/lib/insights-operator/insights-2026-03-20-124208.tar.gz I0320 12:42:08.267279 1 diskrecorder.go:51] Wrote 119 records to disk in 7ms I0320 12:42:08.267309 1 periodic.go:285] Gathering cluster info every 2h0m0s I0320 12:42:08.267324 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0320 12:43:15.841271 1 diskrecorder.go:223] Found files to send: insights-2026-03-20-124208.tar.gz I0320 12:43:15.841300 1 insightsuploader.go:150] Checking archives to upload periodically every 15m23.427457034s I0320 12:43:15.841310 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0320 12:43:15.850674 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0320 12:43:16.003481 1 requests.go:87] Successfully reported id=2026-03-20T12:43:15Z x-rh-insights-request-id=646ad1925d18414fab144609abca4fd8, wrote=58783 I0320 12:43:16.003551 1 insightsuploader.go:187] Uploaded report successfully in 162.231683ms I0320 12:43:16.003578 1 controller.go:129] Initializing last reported time to 2026-03-20T12:43:15Z I0320 12:43:16.003633 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0320 12:43:16.003641 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0320 12:43:16.003649 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0320 12:43:16.008657 1 controller.go:554] The operator is healthy I0320 12:43:45.857074 1 controller.go:554] The operator is healthy I0320 12:44:17.437471 1 insightsreport.go:137] Pulling report from smart-proxy I0320 12:44:17.437495 1 insightsreport.go:149] Retrieving report I0320 12:44:17.444109 1 requests.go:110] Retrieving report for cluster: 6d0aeff8-5003-450c-924a-9ce21390b8c4 I0320 12:44:17.444118 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/6d0aeff8-5003-450c-924a-9ce21390b8c4/reports I0320 12:44:17.447123 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/6d0aeff8-5003-450c-924a-9ce21390b8c4/reports I0320 12:44:17.607042 1 insightsreport.go:184] Report retrieved I0320 12:44:17.632690 1 insightsreport.go:239] Report retrieved correctly I0320 12:45:45.856858 1 controller.go:554] The operator is healthy I0320 12:46:45.834455 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 12:46:45.838844 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 12:46:45.838860 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 12:46:45.842088 1 secretconfigobserver.go:119] support secret does not exist I0320 12:47:45.856527 1 controller.go:554] The operator is healthy I0320 12:49:45.858016 1 controller.go:554] The operator is healthy I0320 12:51:45.842916 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 12:51:45.843256 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 12:51:45.883484 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 12:51:45.883501 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 12:51:45.883580 1 controller.go:554] The operator is healthy I0320 12:51:45.893039 1 secretconfigobserver.go:119] support secret does not exist I0320 12:53:45.856687 1 controller.go:554] The operator is healthy I0320 12:55:45.856895 1 controller.go:554] The operator is healthy I0320 12:56:45.893746 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 12:56:45.902317 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 12:56:45.902334 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 12:56:45.908029 1 secretconfigobserver.go:119] support secret does not exist I0320 12:57:45.856349 1 controller.go:554] The operator is healthy I0320 12:58:44.432719 1 insightsuploader.go:146] Nothing to report since 2026-03-20T12:43:15Z I0320 12:59:45.857138 1 controller.go:554] The operator is healthy I0320 13:01:45.831838 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 13:01:45.856783 1 controller.go:554] The operator is healthy I0320 13:01:45.908705 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 13:01:45.912556 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 13:01:45.912574 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 13:01:45.915917 1 secretconfigobserver.go:119] support secret does not exist I0320 13:03:45.856723 1 controller.go:554] The operator is healthy I0320 13:05:45.860297 1 controller.go:554] The operator is healthy I0320 13:06:45.916813 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 13:06:45.922278 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 13:06:45.922297 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 13:06:45.927689 1 secretconfigobserver.go:119] support secret does not exist I0320 13:07:45.857238 1 controller.go:554] The operator is healthy I0320 13:09:45.856401 1 controller.go:554] The operator is healthy I0320 13:11:45.835624 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 13:11:45.839162 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 13:11:45.855954 1 controller.go:554] The operator is healthy I0320 13:11:45.928093 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 13:11:45.932235 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 13:11:45.932252 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 13:11:45.935541 1 secretconfigobserver.go:119] support secret does not exist