I0216 18:14:25.626485 1 cmd.go:253] Using service-serving-cert provided certificates I0216 18:14:25.627117 1 observer_polling.go:159] Starting file observer I0216 18:14:26.093937 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0216 18:14:26.094247 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0216 18:14:26.094986 1 secure_serving.go:57] Forcing use of http/1.1 only W0216 18:14:26.095011 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0216 18:14:26.095015 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0216 18:14:26.095018 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0216 18:14:26.095021 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0216 18:14:26.095023 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0216 18:14:26.095026 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0216 18:14:26.095319 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0216 18:14:26.099824 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0216 18:14:26.100524 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0216 18:14:26.100578 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0216 18:14:26.100599 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0216 18:14:26.100585 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0216 18:14:26.100633 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0216 18:14:26.100685 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0216 18:14:26.100729 1 secure_serving.go:213] Serving securely on [::]:8443 I0216 18:14:26.100537 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"abe00503-996d-4eda-9527-fc52644d2599", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0216 18:14:26.102173 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0216 18:14:26.102927 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0216 18:14:26.104845 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0216 18:14:26.104897 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0216 18:14:26.105018 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0216 18:14:26.112437 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 18:14:26.112516 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0216 18:14:26.116294 1 secretconfigobserver.go:119] support secret does not exist I0216 18:14:26.120357 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 18:14:26.124315 1 secretconfigobserver.go:119] support secret does not exist I0216 18:14:26.127659 1 recorder.go:161] Pruning old reports every 7h37m47s, max age is 288h0m0s I0216 18:14:26.132649 1 periodic.go:209] Running clusterconfig gatherer I0216 18:14:26.132658 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0216 18:14:26.132656 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0216 18:14:26.132697 1 insightsreport.go:296] Starting report retriever I0216 18:14:26.132702 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0216 18:14:26.132708 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0216 18:14:26.132730 1 tasks_processing.go:45] number of workers: 32 I0216 18:14:26.132750 1 tasks_processing.go:69] worker 1 listening for tasks. I0216 18:14:26.132805 1 tasks_processing.go:69] worker 18 listening for tasks. I0216 18:14:26.132843 1 tasks_processing.go:69] worker 16 listening for tasks. I0216 18:14:26.132871 1 tasks_processing.go:69] worker 29 listening for tasks. I0216 18:14:26.132912 1 tasks_processing.go:71] worker 29 working on tsdb_status task. I0216 18:14:26.132947 1 tasks_processing.go:69] worker 27 listening for tasks. I0216 18:14:26.132980 1 tasks_processing.go:71] worker 27 working on pdbs task. I0216 18:14:26.132832 1 tasks_processing.go:69] worker 23 listening for tasks. I0216 18:14:26.133025 1 tasks_processing.go:71] worker 23 working on openshift_logging task. I0216 18:14:26.132946 1 tasks_processing.go:69] worker 7 listening for tasks. I0216 18:14:26.133102 1 tasks_processing.go:71] worker 7 working on metrics task. I0216 18:14:26.132780 1 tasks_processing.go:69] worker 2 listening for tasks. I0216 18:14:26.132790 1 tasks_processing.go:69] worker 3 listening for tasks. I0216 18:14:26.132798 1 tasks_processing.go:69] worker 4 listening for tasks. I0216 18:14:26.132800 1 tasks_processing.go:69] worker 0 listening for tasks. I0216 18:14:26.132810 1 tasks_processing.go:71] worker 1 working on cluster_apiserver task. I0216 18:14:26.133326 1 tasks_processing.go:71] worker 0 working on sap_config task. I0216 18:14:26.133391 1 tasks_processing.go:71] worker 3 working on dvo_metrics task. I0216 18:14:26.132813 1 tasks_processing.go:69] worker 17 listening for tasks. I0216 18:14:26.133516 1 tasks_processing.go:71] worker 17 working on storage_classes task. I0216 18:14:26.132816 1 tasks_processing.go:69] worker 24 listening for tasks. I0216 18:14:26.132821 1 tasks_processing.go:69] worker 11 listening for tasks. I0216 18:14:26.132821 1 tasks_processing.go:69] worker 21 listening for tasks. I0216 18:14:26.132826 1 tasks_processing.go:69] worker 22 listening for tasks. I0216 18:14:26.132829 1 tasks_processing.go:69] worker 12 listening for tasks. I0216 18:14:26.132830 1 tasks_processing.go:69] worker 19 listening for tasks. I0216 18:14:26.132836 1 tasks_processing.go:69] worker 13 listening for tasks. I0216 18:14:26.132839 1 tasks_processing.go:69] worker 10 listening for tasks. I0216 18:14:26.132836 1 tasks_processing.go:69] worker 20 listening for tasks. I0216 18:14:26.132849 1 tasks_processing.go:69] worker 5 listening for tasks. I0216 18:14:26.132853 1 tasks_processing.go:71] worker 18 working on active_alerts task. I0216 18:14:26.132856 1 tasks_processing.go:69] worker 8 listening for tasks. I0216 18:14:26.132853 1 tasks_processing.go:69] worker 14 listening for tasks. I0216 18:14:26.132862 1 tasks_processing.go:69] worker 25 listening for tasks. I0216 18:14:26.132858 1 tasks_processing.go:69] worker 15 listening for tasks. I0216 18:14:26.132863 1 tasks_processing.go:69] worker 9 listening for tasks. I0216 18:14:26.133398 1 tasks_processing.go:71] worker 2 working on machines task. I0216 18:14:26.133877 1 tasks_processing.go:71] worker 13 working on sap_datahubs task. I0216 18:14:26.133928 1 tasks_processing.go:71] worker 21 working on nodenetworkconfigurationpolicies task. I0216 18:14:26.132880 1 tasks_processing.go:69] worker 28 listening for tasks. I0216 18:14:26.134071 1 tasks_processing.go:71] worker 28 working on clusterroles task. I0216 18:14:26.134321 1 tasks_processing.go:71] worker 22 working on node_logs task. I0216 18:14:26.132883 1 tasks_processing.go:71] worker 16 working on container_images task. I0216 18:14:26.134526 1 tasks_processing.go:71] worker 11 working on networks task. I0216 18:14:26.133881 1 tasks_processing.go:71] worker 9 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0216 18:14:26.132879 1 tasks_processing.go:69] worker 6 listening for tasks. I0216 18:14:26.134921 1 tasks_processing.go:71] worker 6 working on openstack_version task. I0216 18:14:26.132882 1 tasks_processing.go:69] worker 30 listening for tasks. I0216 18:14:26.132891 1 tasks_processing.go:69] worker 26 listening for tasks. I0216 18:14:26.132772 1 tasks_processing.go:69] worker 31 listening for tasks. I0216 18:14:26.133423 1 tasks_processing.go:71] worker 4 working on machine_config_pools task. I0216 18:14:26.135119 1 tasks_processing.go:71] worker 31 working on version task. I0216 18:14:26.133891 1 tasks_processing.go:71] worker 10 working on openstack_controlplanes task. I0216 18:14:26.135231 1 tasks_processing.go:71] worker 30 working on monitoring_persistent_volumes task. I0216 18:14:26.135265 1 tasks_processing.go:71] worker 26 working on qemu_kubevirt_launcher_logs task. I0216 18:14:26.133895 1 tasks_processing.go:71] worker 20 working on oauths task. I0216 18:14:26.135608 1 tasks_processing.go:71] worker 23 working on support_secret task. I0216 18:14:26.133899 1 tasks_processing.go:71] worker 5 working on machine_sets task. I0216 18:14:26.133905 1 tasks_processing.go:71] worker 8 working on validating_webhook_configurations task. I0216 18:14:26.133908 1 tasks_processing.go:71] worker 14 working on container_runtime_configs task. I0216 18:14:26.133911 1 tasks_processing.go:71] worker 25 working on feature_gates task. I0216 18:14:26.133916 1 tasks_processing.go:71] worker 15 working on crds task. I0216 18:14:26.134498 1 tasks_processing.go:71] worker 19 working on pod_network_connectivity_checks task. I0216 18:14:26.134503 1 tasks_processing.go:71] worker 12 working on image task. I0216 18:14:26.134515 1 tasks_processing.go:71] worker 24 working on jaegers task. I0216 18:14:26.135654 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 2.571015ms to process 0 records I0216 18:14:26.137179 1 tasks_processing.go:71] worker 2 working on nodenetworkstates task. E0216 18:14:26.137239 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0216 18:14:26.137283 1 gather.go:177] gatherer "clusterconfig" function "machines" took 3.410061ms to process 0 records I0216 18:14:26.141223 1 tasks_processing.go:71] worker 21 working on openstack_dataplanenodesets task. I0216 18:14:26.141249 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 7.242463ms to process 0 records I0216 18:14:26.141372 1 tasks_processing.go:71] worker 4 working on sap_pods task. I0216 18:14:26.141376 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 6.417566ms to process 0 records I0216 18:14:26.141743 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0216 18:14:26.141767 1 controller.go:326] The initial operator extension status is healthy I0216 18:14:26.141779 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0216 18:14:26.141787 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0216 18:14:26.141792 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0216 18:14:26.141813 1 controller.go:489] The operator is still being initialized I0216 18:14:26.141811 1 tasks_processing.go:71] worker 1 working on cost_management_metrics_configs task. I0216 18:14:26.141821 1 controller.go:512] The operator is healthy I0216 18:14:26.141962 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0216 18:14:26.142043 1 operator.go:288] started I0216 18:14:26.142070 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0216 18:14:26.142088 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0216 18:14:26.142151 1 recorder.go:75] Recording config/apiserver with fingerprint=afa270c03515d2c5e81339cec3bc90abcfcdf8d09f1d839281b36fbe6d9f3416 I0216 18:14:26.142175 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 8.497002ms to process 1 records E0216 18:14:26.144058 1 gather_active_alerts.go:64] Unable to retrieve most recent alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.129.0.9:48557->172.30.0.10:53: read: connection refused I0216 18:14:26.144083 1 tasks_processing.go:71] worker 18 working on ingress task. E0216 18:14:26.144106 1 gather.go:140] gatherer "clusterconfig" function "active_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.129.0.9:48557->172.30.0.10:53: read: connection refused I0216 18:14:26.144176 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 10.500042ms to process 0 records I0216 18:14:26.144363 1 tasks_processing.go:71] worker 5 working on install_plans task. I0216 18:14:26.144385 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 8.619271ms to process 0 records E0216 18:14:26.144539 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0216 18:14:26.144587 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 8.468025ms to process 0 records I0216 18:14:26.144615 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 8.412895ms to process 0 records I0216 18:14:26.144682 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 10.521788ms to process 0 records I0216 18:14:26.144713 1 tasks_processing.go:71] worker 13 working on lokistack task. I0216 18:14:26.144551 1 tasks_processing.go:71] worker 19 working on machine_autoscalers task. E0216 18:14:26.144912 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.129.0.9:32874->172.30.0.10:53: read: connection refused E0216 18:14:26.144921 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.129.0.9:32874->172.30.0.10:53: read: connection refused I0216 18:14:26.144719 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 7.222545ms to process 0 records I0216 18:14:26.144705 1 tasks_processing.go:71] worker 24 working on mutating_webhook_configurations task. I0216 18:14:26.144999 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 9.354967ms to process 0 records I0216 18:14:26.145015 1 tasks_processing.go:71] worker 10 working on authentication task. I0216 18:14:26.144729 1 tasks_processing.go:71] worker 2 working on olm_operators task. I0216 18:14:26.145170 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=9670387bc3b90e4e945959eecee990ccc1e6eed7270f220f9bea25bdc431a506 I0216 18:14:26.145195 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=11e6723e907d0a04e81d6228dc7ebfa156486fa9f858a2a71a7009d5f4dea8ad I0216 18:14:26.145215 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=b04a86bde316625e65df3e721f690ebd33fdc5093df6518e1544b34f5c5a4953 I0216 18:14:26.145239 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=1db3b2f34d76bc55f2aef9224798c0b0547185e8254f51a5e8af2c49af6276d7 I0216 18:14:26.145245 1 tasks_processing.go:71] worker 27 working on image_registries task. I0216 18:14:26.145292 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=fc4b2a7e57f68cc71b658876762686776c2c1f24fa6a6651fa926223e398c9d5 I0216 18:14:26.145445 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=c8c9e3260f6385ea91a3120676552405f5c9829c8508a2e92502e1df03a198db I0216 18:14:26.145487 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 11.530041ms to process 6 records I0216 18:14:26.145512 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 11.207446ms to process 0 records I0216 18:14:26.145540 1 tasks_processing.go:71] worker 0 working on machine_configs task. I0216 18:14:26.145595 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=443fc5c0fec31a30af4cbbcb7fb3b75efc3fb79ed1d6773b7def94481470a967 I0216 18:14:26.145618 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=4f5baae6de613a363a83bdd1864ef53e61f38f1fe5a72af35f7791cae9a153df I0216 18:14:26.145630 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 11.024911ms to process 2 records E0216 18:14:26.145639 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.129.0.9:32874->172.30.0.10:53: read: connection refused I0216 18:14:26.145649 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 11.990845ms to process 0 records I0216 18:14:26.145698 1 tasks_processing.go:71] worker 17 working on silenced_alerts task. I0216 18:14:26.145714 1 tasks_processing.go:71] worker 29 working on ingress_certificates task. E0216 18:14:26.145656 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.129.0.9:32874->172.30.0.10:53: read: connection refused I0216 18:14:26.145735 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 11.829084ms to process 0 records I0216 18:14:26.145846 1 tasks_processing.go:71] worker 7 working on storage_cluster task. I0216 18:14:26.146348 1 tasks_processing.go:71] worker 14 working on schedulers task. I0216 18:14:26.146367 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 10.528467ms to process 0 records I0216 18:14:26.146378 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 11.440197ms to process 0 records I0216 18:14:26.146456 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 5.171424ms to process 0 records I0216 18:14:26.146504 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 4.677819ms to process 0 records I0216 18:14:26.146522 1 tasks_processing.go:71] worker 6 working on openshift_machine_api_events task. I0216 18:14:26.146526 1 tasks_processing.go:71] worker 1 working on operators_pods_and_events task. I0216 18:14:26.146538 1 tasks_processing.go:71] worker 21 working on certificate_signing_requests task. I0216 18:14:26.146639 1 tasks_processing.go:71] worker 4 working on image_pruners task. I0216 18:14:26.146729 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 5.256034ms to process 0 records E0216 18:14:26.151937 1 gather_silenced_alerts.go:52] Unable to retrieve silenced alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.129.0.9:47729->172.30.0.10:53: read: connection refused I0216 18:14:26.152011 1 tasks_processing.go:71] worker 17 working on machine_healthchecks task. E0216 18:14:26.152304 1 gather.go:140] gatherer "clusterconfig" function "silenced_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.129.0.9:47729->172.30.0.10:53: read: connection refused I0216 18:14:26.152362 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 6.295598ms to process 0 records I0216 18:14:26.159295 1 tasks_processing.go:71] worker 13 working on config_maps task. I0216 18:14:26.159366 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 14.514396ms to process 0 records I0216 18:14:26.159406 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0216 18:14:26.159449 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0216 18:14:26.159475 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0216 18:14:26.159500 1 controller.go:212] Source scaController *sca.Controller is not ready I0216 18:14:26.159538 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0216 18:14:26.159589 1 controller.go:489] The operator is still being initialized I0216 18:14:26.159619 1 controller.go:512] The operator is healthy I0216 18:14:26.161197 1 tasks_processing.go:71] worker 12 working on ceph_cluster task. I0216 18:14:26.162579 1 recorder.go:75] Recording config/image with fingerprint=ac6cbead975ba18091ae6aadcf3d538711f49243e4becc7cef7a2379155fa69f I0216 18:14:26.162652 1 gather.go:177] gatherer "clusterconfig" function "image" took 25.176323ms to process 1 records I0216 18:14:26.162732 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 15.95367ms to process 0 records I0216 18:14:26.162793 1 tasks_processing.go:71] worker 7 working on openstack_dataplanedeployments task. I0216 18:14:26.162835 1 tasks_processing.go:71] worker 23 working on infrastructures task. I0216 18:14:26.162601 1 gather_logs.go:145] no pods in namespace were found E0216 18:14:26.162796 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0216 18:14:26.164382 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 26.149698ms to process 0 records I0216 18:14:26.164422 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 17.326877ms to process 0 records I0216 18:14:26.164481 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 27.00786ms to process 0 records I0216 18:14:26.165006 1 recorder.go:75] Recording config/network with fingerprint=38ec3b8cebe91814f6d8fb096bf6fd113bba0affeed2b10fe110cb7189dd0a2b I0216 18:14:26.165064 1 gather.go:177] gatherer "clusterconfig" function "networks" took 27.894545ms to process 1 records I0216 18:14:26.165107 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 28.078719ms to process 0 records I0216 18:14:26.165299 1 recorder.go:75] Recording config/featuregate with fingerprint=d10b9d1e1584773398878c7bdf335cb1f29dd176b9e67a1f6181ab6861c791a5 I0216 18:14:26.165338 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 26.661032ms to process 1 records I0216 18:14:26.165888 1 recorder.go:75] Recording config/oauth with fingerprint=b3aa4934ebda8b227bc658ead25e8a1c9cc6609f02f7c0f419bf4a16b0de87a3 I0216 18:14:26.165949 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 27.217063ms to process 1 records I0216 18:14:26.166687 1 tasks_processing.go:71] worker 11 working on nodes task. I0216 18:14:26.167516 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=2ce92afd546b97f0e50954e0e2f9bc85e0ecded76833a010748662600530bc3a I0216 18:14:26.167570 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 18.925791ms to process 1 records I0216 18:14:26.167602 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 28.926303ms to process 0 records I0216 18:14:26.168006 1 tasks_processing.go:74] worker 26 stopped. I0216 18:14:26.168026 1 tasks_processing.go:71] worker 19 working on overlapping_namespace_uids task. I0216 18:14:26.168521 1 tasks_processing.go:74] worker 27 stopped. I0216 18:14:26.168835 1 tasks_processing.go:71] worker 25 working on proxies task. I0216 18:14:26.169583 1 tasks_processing.go:71] worker 20 working on operators task. I0216 18:14:26.169841 1 tasks_processing.go:74] worker 28 stopped. I0216 18:14:26.169882 1 tasks_processing.go:71] worker 22 working on service_accounts task. I0216 18:14:26.169896 1 tasks_processing.go:71] worker 30 working on aggregated_monitoring_cr_names task. I0216 18:14:26.170591 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=16cd07ad95a64b258e31e46b467c61f96babf9dac423849eac081462ef8a470e I0216 18:14:26.170823 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=1057d5b9c73df675f63920b04766ffaec4981fe614ea3ea08ea024550d36bcab I0216 18:14:26.170873 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 35.758771ms to process 2 records I0216 18:14:26.171099 1 recorder.go:75] Recording config/ingress with fingerprint=ef096b394c21f6b8f5394a9f55828ea14dff308dba00041ef0232d6d85b63488 I0216 18:14:26.171134 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 26.175235ms to process 1 records E0216 18:14:26.171163 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0216 18:14:26.171193 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 18.689557ms to process 0 records I0216 18:14:26.171221 1 tasks_processing.go:74] worker 17 stopped. I0216 18:14:26.171245 1 tasks_processing.go:74] worker 18 stopped. I0216 18:14:26.171375 1 tasks_processing.go:74] worker 10 stopped. I0216 18:14:26.171710 1 recorder.go:75] Recording config/authentication with fingerprint=612ba0f078cf02ea840a3f30c3270c27583050921a54aa03c9116b9a63490cc8 I0216 18:14:26.171768 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 26.34473ms to process 1 records I0216 18:14:26.171799 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 24.928952ms to process 0 records I0216 18:14:26.171895 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=f4eed953c97f4985b1fc9cac176b870c91a3ddfeb81663405d50d47d8393b2dc I0216 18:14:26.171927 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 25.349778ms to process 1 records I0216 18:14:26.171955 1 tasks_processing.go:74] worker 14 stopped. I0216 18:14:26.171976 1 tasks_processing.go:74] worker 12 stopped. I0216 18:14:26.171997 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 10.710103ms to process 0 records I0216 18:14:26.171979 1 tasks_processing.go:74] worker 6 stopped. I0216 18:14:26.172133 1 tasks_processing.go:74] worker 4 stopped. I0216 18:14:26.172263 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=6331f73fe2e5585eaa3c6414d59d2a821947f3979507562359e0d4ca0cbe109c I0216 18:14:26.172287 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 25.476243ms to process 1 records E0216 18:14:26.172463 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%2717e28ab1-5431-4310-a2bb-bd9f61798466%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.9:54873->172.30.0.10:53: read: connection refused I0216 18:14:26.172505 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%2717e28ab1-5431-4310-a2bb-bd9f61798466%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.9:54873->172.30.0.10:53: read: connection refused I0216 18:14:26.172994 1 tasks_processing.go:74] worker 15 stopped. I0216 18:14:26.173714 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=864731cfb6cb027c24dd8a627f5836baa743b09ea1cd5e808bdb73db136e8b94 I0216 18:14:26.174149 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=4b9e5cf4d639c43400ab1cbd7fc0e329f9bd7c12cad83614a1b89c462b67c47a I0216 18:14:26.174213 1 gather.go:177] gatherer "clusterconfig" function "crds" took 36.937091ms to process 2 records I0216 18:14:26.175596 1 tasks_processing.go:74] worker 21 stopped. I0216 18:14:26.175687 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 29.045793ms to process 0 records I0216 18:14:26.177788 1 tasks_processing.go:74] worker 7 stopped. I0216 18:14:26.177948 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 14.972136ms to process 0 records I0216 18:14:26.181086 1 tasks_processing.go:74] worker 2 stopped. I0216 18:14:26.181220 1 recorder.go:75] Recording config/olm_operators with fingerprint=1614482ef31b0b00ebf60aea6ac0bd3f762ea536b474c2282ffc752723fb6d90 I0216 18:14:26.181257 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 35.91479ms to process 1 records I0216 18:14:26.182091 1 tasks_processing.go:74] worker 19 stopped. I0216 18:14:26.182161 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0216 18:14:26.182196 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 14.056869ms to process 1 records I0216 18:14:26.183017 1 recorder.go:75] Recording config/infrastructure with fingerprint=6b86eab341545e68f51b81b45ec87c063d431817e84945f5ba5ad05ce234fbd4 I0216 18:14:26.183072 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 19.344729ms to process 1 records I0216 18:14:26.183098 1 tasks_processing.go:74] worker 23 stopped. W0216 18:14:26.183143 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0216 18:14:26.183176 1 tasks_processing.go:74] worker 25 stopped. I0216 18:14:26.183169 1 recorder.go:75] Recording config/proxy with fingerprint=31fe3984b6f2d8997e87452fe6f57b5afb43efa62b71d22dee7856d8dc2ee4c7 I0216 18:14:26.183235 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 14.122099ms to process 1 records I0216 18:14:26.183378 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0216 18:14:26.187573 1 tasks_processing.go:74] worker 16 stopped. W0216 18:14:26.190471 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.9:40483->172.30.0.10:53: read: connection refused I0216 18:14:26.190489 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.9:40483->172.30.0.10:53: read: connection refused I0216 18:14:26.191098 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-smtwj with fingerprint=b4a40e1c51222ac3519022855859ac543e71611598c8cfad432167b48ca9b22a I0216 18:14:26.191495 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-p4f8s with fingerprint=dccf86ee2802802d325d962201a9b55250b13105ae93447be7e1186ca45691f5 I0216 18:14:26.191557 1 recorder.go:75] Recording config/running_containers with fingerprint=d181f9075ceae07fc60f14032bcffbea704c044ca0132d9a191370d5cdd1521d I0216 18:14:26.191574 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 53.036839ms to process 3 records I0216 18:14:26.191588 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 55.799159ms to process 0 records I0216 18:14:26.191597 1 tasks_processing.go:74] worker 9 stopped. I0216 18:14:26.195114 1 tasks_processing.go:74] worker 30 stopped. I0216 18:14:26.195141 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 24.685867ms to process 0 records I0216 18:14:26.196002 1 tasks_processing.go:74] worker 8 stopped. I0216 18:14:26.196225 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=8e7c526b19febff8d1d7a74b022d87153f6f5b7d80069880739d563983aadc15 I0216 18:14:26.196325 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=3fc7461153dba9899566998b3ef28b443daeef7fb90ba22495f973d523dfad64 I0216 18:14:26.196355 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=3a8c77f5ba27a50801b4c4d7f270bd07210ab078d6c4eb2aa4fab75cec3c92f9 I0216 18:14:26.196386 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=d04468cecb609cb9d90ff2d45f2e55401267ab9fac21e718884d5f2716bf39eb I0216 18:14:26.196417 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=9bee3657ad4d057c082331c101bdfa622d998521b194b6a1c8f8d49adcf865b6 I0216 18:14:26.196448 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=952929e8749e2cb73400d0df7faba64545115b53cacc3eff391e38623aa4881a I0216 18:14:26.196489 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=332bc5703fbe4775fb1b776ddd01c9ca7a9d9d30793e8d5adafdfeeee0846952 I0216 18:14:26.196530 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=7028bf58b043949e0b98f3fc938517d4121bcd49ff14602c88a1bdb1ebcc9820 I0216 18:14:26.196560 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=47a29a77f889b75159bb8f837b0271150f17538c70111a486c9fcedca6f82e22 I0216 18:14:26.196590 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=b37d64a6403992ac76b1cf591312ad0a98c46b8f7e987bb11b9d95ba1fe75955 I0216 18:14:26.196619 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=26a9d9ef0552beaa09bf95c2e1c47fa7375d20877b0c6e7460f3947fc96004d9 I0216 18:14:26.196627 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 60.053716ms to process 11 records I0216 18:14:26.197351 1 tasks_processing.go:74] worker 24 stopped. I0216 18:14:26.197463 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=bda7473463bc76084133db5c87a7e3e511c94121fd3596ac8a9ad673d4f22177 I0216 18:14:26.197500 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=a4c8600357123a0901f26a487dca3e64cb3fffac5d50120fe846088612619a82 I0216 18:14:26.197529 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=bfaa38149094294b0cdba33a119a5dae923d356c84e8341fb758970c2f85cce3 I0216 18:14:26.197538 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 52.34098ms to process 3 records I0216 18:14:26.200164 1 tasks_processing.go:74] worker 29 stopped. E0216 18:14:26.200176 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0216 18:14:26.200183 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2oggm30vmofomplkfldgt27943ql796q-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2oggm30vmofomplkfldgt27943ql796q-primary-cert-bundle-secret" not found I0216 18:14:26.200233 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=51348f12bbb984ab3cfd542c660ae81953105f7685ef3b58ff09122e15a34592 I0216 18:14:26.200245 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 54.435067ms to process 1 records I0216 18:14:26.201192 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0216 18:14:26.201201 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0216 18:14:26.201212 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0216 18:14:26.202079 1 tasks_processing.go:74] worker 13 stopped. E0216 18:14:26.202095 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0216 18:14:26.202102 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0216 18:14:26.202107 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0216 18:14:26.202119 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0216 18:14:26.202158 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0216 18:14:26.202169 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0216 18:14:26.202178 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0216 18:14:26.202191 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0216 18:14:26.202241 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0216 18:14:26.202251 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0216 18:14:26.202256 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 42.755986ms to process 7 records I0216 18:14:26.205553 1 base_controller.go:82] Caches are synced for ConfigController I0216 18:14:26.205567 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0216 18:14:26.209126 1 configmapobserver.go:84] configmaps "insights-config" not found I0216 18:14:26.242865 1 base_controller.go:82] Caches are synced for LoggingSyncer I0216 18:14:26.242936 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0216 18:14:26.253763 1 tasks_processing.go:74] worker 31 stopped. I0216 18:14:26.254138 1 recorder.go:75] Recording config/version with fingerprint=4558e8a84c9c981e3bde87d57287fe924e026e06c63f9debee6353f938a05e1b I0216 18:14:26.254163 1 recorder.go:75] Recording config/id with fingerprint=fecc0832a6735d1248045cfa741194359954eaa18b1c5d670103f29f37391a10 I0216 18:14:26.254174 1 gather.go:177] gatherer "clusterconfig" function "version" took 118.621987ms to process 2 records I0216 18:14:26.254252 1 tasks_processing.go:74] worker 11 stopped. I0216 18:14:26.254552 1 recorder.go:75] Recording config/node/ip-10-0-1-84.ec2.internal with fingerprint=9db852b55c86c612e2ea6607c204d49cce0f7edcf041ac9dc7da9d65eb60f55b I0216 18:14:26.254693 1 recorder.go:75] Recording config/node/ip-10-0-2-109.ec2.internal with fingerprint=de756106f042137cc3b5c5816158ed07ba3e575a32d72cfb6a172bd2c6bf7ab4 I0216 18:14:26.254740 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 87.045349ms to process 2 records I0216 18:14:26.256760 1 tasks_processing.go:74] worker 0 stopped. I0216 18:14:26.256794 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0216 18:14:26.256808 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 111.197628ms to process 1 records W0216 18:14:27.167098 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0216 18:14:27.789259 1 gather_cluster_operator_pods_and_events.go:121] Found 24 pods with 26 containers I0216 18:14:27.789276 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 967916 bytes I0216 18:14:27.789329 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for networking-console-plugin container networking-console-plugin-6ddbfdf749-p5cwd pod in namespace openshift-network-console (previous: false). I0216 18:14:28.004237 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0216 18:14:28.019137 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for networking-console-plugin container networking-console-plugin-6ddbfdf749-ssjv9 pod in namespace openshift-network-console (previous: false). W0216 18:14:28.166441 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0216 18:14:28.215112 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console-operator container console-operator-575cd97545-smtwj pod in namespace openshift-console-operator (previous: true). I0216 18:14:28.396446 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console-operator container console-operator-575cd97545-smtwj pod in namespace openshift-console-operator (previous: false). I0216 18:14:28.602883 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console container console-646bd5866-2fjt5 pod in namespace openshift-console (previous: false). I0216 18:14:28.605690 1 tasks_processing.go:74] worker 20 stopped. I0216 18:14:28.605811 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=9a210034bdb47cee9809297e5698ac9fa4808dbf859e111e943373149af83111 I0216 18:14:28.605876 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=b1d3a57201f472fba248e6e22732d03f4ebac77f3d7204455295e9c6f31f376a I0216 18:14:28.605902 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=34e630c2d35f99c1230d123c36b0c8462b04734eabd40b682027d283281b985b I0216 18:14:28.605912 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0216 18:14:28.605934 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=c944dc9c80fd645d5dd69fd6479abe98e8a019b0d22c5ac098bff586da3280ea I0216 18:14:28.605950 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0216 18:14:28.605972 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=eac69ef94030117e32c211eec8529d8d593804ed2d1fd5538738e84211f80de0 I0216 18:14:28.606000 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=7d58fbd906be163957ac8b4ef2a46447f3a156bbb7e67bab50cb3633a6666425 I0216 18:14:28.606023 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=15d2eb4a80ab9eeb0567da1d35bc5b795746f037af18a1ded97ce8083d5dd42f I0216 18:14:28.606054 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=e0030f23bb33bee37725a6b35a4271bd55b7bd8adc801dedac707e401fc0b96c I0216 18:14:28.606296 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=41a269385e6ef6312ca3f558100a258f8e8d55bcefc08d476056f65a41a1c43b I0216 18:14:28.606315 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=0563512c09724f2d518aaacd6b210c2fd2b18f162f6ad7c64cf2c9cf3e8236d0 I0216 18:14:28.606324 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0216 18:14:28.606339 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=89f036461167bcbf85bb70187b512f89a025e83ca1b5363ca05fcbe4f2c4ddc1 I0216 18:14:28.606349 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0216 18:14:28.606364 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=0382a5d00e37795bc4dfef1337471a4219639a7c0ea70182e75ee26bb5e3671c I0216 18:14:28.606373 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0216 18:14:28.606394 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=d5aef549a94c8c7fed1683f381d1c201801157fd12b72fa25c4e5698d159c7a4 I0216 18:14:28.606403 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0216 18:14:28.606423 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=5c0c1d8f76fd50144cdd871c61820393ab1f3d95b150ba4eacd0f61084f32461 I0216 18:14:28.606523 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=9c333dc508338ebf0127a94b0c381832bbf586b3b28749231a30cf30a991fecc I0216 18:14:28.606532 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0216 18:14:28.606540 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0216 18:14:28.606561 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0216 18:14:28.606584 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=53639c2abf57893d4972d67d7e35aa235427f3b4d90b67d613407cf8cd166137 I0216 18:14:28.606608 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=b19be015df66af03ee7fe2b28702f6a8e683f939bdd31b695054e0eb58610888 I0216 18:14:28.606618 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0216 18:14:28.606633 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=7fa65c96ff60c34feb9970392b4d1b35775b04dd7c6d55244e4d5ebb59aefc98 I0216 18:14:28.606643 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0216 18:14:28.606655 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=cc94c14d5f5636d6d8be2076c6009ca6b2d63fbc29d40c8748b9db59924b0839 I0216 18:14:28.606694 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=1287fa79f5026ad4ed98afc71a202e650c17d58f20beb51956fa43986216c6e9 I0216 18:14:28.606712 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=e078231d298d86a822cd0be602dc78b5a9c20e93549848f3194ee0c5b810ca4b I0216 18:14:28.606730 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=a11491cd3b635572e4d185bb083906b832b98c7da3ae18008e8392e09288995b I0216 18:14:28.606751 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=64b136c686108014b307b0ac7ffd29269ebca9a5f9ff9b520ba1630daeb4fe0e I0216 18:14:28.606761 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0216 18:14:28.606795 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=07b0798807567905b78da9a23862935aa5988ba1a576d3b15add1a79cde6c169 I0216 18:14:28.606812 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0216 18:14:28.606818 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0216 18:14:28.606825 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.436056015s to process 38 records I0216 18:14:28.789431 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0216 18:14:28.789450 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console container console-646bd5866-kzmp4 pod in namespace openshift-console (previous: false). I0216 18:14:28.989336 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0216 18:14:28.989354 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console container console-654d7754f4-l2r56 pod in namespace openshift-console (previous: false). W0216 18:14:29.166415 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0216 18:14:29.197453 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console container console-654d7754f4-sqbzd pod in namespace openshift-console (previous: false). I0216 18:14:29.395425 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for console-654d7754f4-sqbzd pod in namespace openshift-console for failing operator console (previous: false): "container \"console\" in pod \"console-654d7754f4-sqbzd\" is terminated" I0216 18:14:29.395443 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"console\" in pod \"console-654d7754f4-sqbzd\" is terminated" I0216 18:14:29.395455 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for download-server container downloads-56cb7c9577-45527 pod in namespace openshift-console (previous: false). I0216 18:14:29.596806 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0216 18:14:29.596826 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for download-server container downloads-56cb7c9577-cl6mr pod in namespace openshift-console (previous: false). I0216 18:14:29.794617 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for downloads-56cb7c9577-cl6mr pod in namespace openshift-console for failing operator download-server (previous: false): "container \"download-server\" in pod \"downloads-56cb7c9577-cl6mr\" is waiting to start: ContainerCreating" I0216 18:14:29.794635 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"download-server\" in pod \"downloads-56cb7c9577-cl6mr\" is waiting to start: ContainerCreating" I0216 18:14:29.794683 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-jg7sl pod in namespace openshift-dns (previous: false). I0216 18:14:29.996875 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-jg7sl pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-jg7sl\" is waiting to start: ContainerCreating" I0216 18:14:29.996894 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-jg7sl\" is waiting to start: ContainerCreating" I0216 18:14:29.996903 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-jg7sl pod in namespace openshift-dns (previous: false). W0216 18:14:30.166599 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0216 18:14:30.198621 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-kr85l pod in namespace openshift-dns (previous: false). I0216 18:14:30.395177 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-kr85l pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-kr85l\" is waiting to start: ContainerCreating" I0216 18:14:30.395197 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-kr85l\" is waiting to start: ContainerCreating" I0216 18:14:30.395205 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-kr85l pod in namespace openshift-dns (previous: false). I0216 18:14:30.595721 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-kr85l pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-kr85l\" is waiting to start: ContainerCreating" I0216 18:14:30.595741 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-kr85l\" is waiting to start: ContainerCreating" I0216 18:14:30.595755 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-ccgql pod in namespace openshift-dns (previous: false). I0216 18:14:30.795234 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0216 18:14:30.795259 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-jtmnx pod in namespace openshift-dns (previous: false). I0216 18:14:30.996477 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0216 18:14:30.996529 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-58c6774647-lmtxb pod in namespace openshift-image-registry (previous: false). W0216 18:14:31.162740 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0216 18:14:31.162770 1 tasks_processing.go:74] worker 3 stopped. E0216 18:14:31.162791 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0216 18:14:31.162801 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0216 18:14:31.162813 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.029309179s to process 1 records I0216 18:14:31.198291 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-58c6774647-lmtxb pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-58c6774647-lmtxb\" is terminated" I0216 18:14:31.198309 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-58c6774647-lmtxb\" is terminated" I0216 18:14:31.198342 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-58c6774647-vzcnt pod in namespace openshift-image-registry (previous: false). I0216 18:14:31.398804 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-66c49b4cfb-26grf pod in namespace openshift-image-registry (previous: false). I0216 18:14:31.590173 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0216 18:14:31.590196 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-66c49b4cfb-wdl8n pod in namespace openshift-image-registry (previous: false). I0216 18:14:31.795924 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0216 18:14:31.795946 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-8nvmd pod in namespace openshift-image-registry (previous: false). I0216 18:14:31.996620 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0216 18:14:31.996640 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-p9gxs pod in namespace openshift-image-registry (previous: false). I0216 18:14:32.206022 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0216 18:14:32.206042 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-56bf9967bf-j54ts pod in namespace openshift-ingress (previous: false). I0216 18:14:32.416059 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-56bf9967bf-s6rgl pod in namespace openshift-ingress (previous: false). I0216 18:14:32.601136 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7548757bc-hgtwt pod in namespace openshift-ingress (previous: false). I0216 18:14:32.796425 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-28wrb pod in namespace openshift-ingress-canary (previous: false). I0216 18:14:32.995060 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-28wrb pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-28wrb\" is waiting to start: ContainerCreating" I0216 18:14:32.995079 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-28wrb\" is waiting to start: ContainerCreating" I0216 18:14:32.995110 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-z95xg pod in namespace openshift-ingress-canary (previous: false). I0216 18:14:33.196584 1 tasks_processing.go:74] worker 1 stopped. I0216 18:14:33.196708 1 recorder.go:75] Recording events/openshift-network-console with fingerprint=514dff6453365cb377034a4e423f2939efda697d9503f9e4d9f077e33688784f I0216 18:14:33.196830 1 recorder.go:75] Recording events/openshift-console-operator with fingerprint=dea69057a8f628f2e2c385efa47ccf056abdb90f790c3017fca5ce053a2fff98 I0216 18:14:33.196878 1 recorder.go:75] Recording events/openshift-console with fingerprint=4cbbe07f629163bd3094ae264ea9a802e76299528e3f1b13a2fe2dd6ceba139d I0216 18:14:33.196905 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=635a36f90573c2f88856acb720cac349818199ae8d1b31017b16485d474fe3db I0216 18:14:33.196945 1 recorder.go:75] Recording events/openshift-dns with fingerprint=5aead4d24f1f5e57d0cb9560c4b6789deea5c63eb5f34a47b9f0bc9c805bb99f I0216 18:14:33.197037 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=e12c6049d56008906b4cef1d67f1318ddab7cf2dc55f1fb62a5de8e559cb3fae I0216 18:14:33.197064 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=2fd30da6daf15cf2bd25735ccb58c789730815a8cbe87dbcfab2281b9fa547ef I0216 18:14:33.197137 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=aecc94860597cac3d5461c12a322902c4e82645ad992ab40b3d758d67eb91335 I0216 18:14:33.197159 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=5542419908387145557fbbf7005d546fde7722262ac34304501fe1cf81438ae9 I0216 18:14:33.197171 1 recorder.go:75] Recording config/pod/openshift-network-console/logs/networking-console-plugin-6ddbfdf749-p5cwd/networking-console-plugin_current.log with fingerprint=bb099244cdfdb412aadc81f629e715d91a8f0b69bbee8fa29f508fd509b6a7be I0216 18:14:33.197178 1 recorder.go:75] Recording config/pod/openshift-network-console/logs/networking-console-plugin-6ddbfdf749-ssjv9/networking-console-plugin_current.log with fingerprint=07eed83df73339104364783ee8e32478b437dce51d898f3294192679a348dc5a I0216 18:14:33.197328 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-smtwj with fingerprint=b4a40e1c51222ac3519022855859ac543e71611598c8cfad432167b48ca9b22a E0216 18:14:33.197346 1 gather.go:161] error recording gatherer "clusterconfig" function "operators_pods_and_events" result "config/pod/openshift-console-operator/console-operator-575cd97545-smtwj.json" because of the error: the record with the same name "config/pod/openshift-console-operator/console-operator-575cd97545-smtwj.json" was already recorded and had the fingerprint "b4a40e1c51222ac3519022855859ac543e71611598c8cfad432167b48ca9b22a", overwriting with the record having fingerprint "b4a40e1c51222ac3519022855859ac543e71611598c8cfad432167b48ca9b22a" W0216 18:14:33.197358 1 gather.go:155] issue recording gatherer "clusterconfig" function "operators_pods_and_events" result "config/pod/openshift-console-operator/console-operator-575cd97545-smtwj.json" because of the warning: warning: the record with the same fingerprint "b4a40e1c51222ac3519022855859ac543e71611598c8cfad432167b48ca9b22a" was already recorded at path "config/pod/openshift-console-operator/console-operator-575cd97545-smtwj.json", recording another one with a different path "config/pod/openshift-console-operator/console-operator-575cd97545-smtwj.json" I0216 18:14:33.197369 1 recorder.go:75] Recording config/pod/openshift-console-operator/logs/console-operator-575cd97545-smtwj/console-operator_previous.log with fingerprint=39399fd36d185c2caee551d0ecf91a476e1a7898a99b1958ffbaf46300ee1e75 I0216 18:14:33.197581 1 recorder.go:75] Recording config/pod/openshift-console-operator/logs/console-operator-575cd97545-smtwj/console-operator_current.log with fingerprint=d9de4136021df35acd9c1e6fc7e6512c643c0f160c3eb5c8e602cf87304e295a I0216 18:14:33.197595 1 recorder.go:75] Recording config/pod/openshift-console/logs/console-654d7754f4-l2r56/console_current.log with fingerprint=15dc5278a62e2ce25e4cbab0e7654f8df11dec0f1241369830e88de655b6ba07 I0216 18:14:33.197721 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-jg7sl with fingerprint=0878282d2c8f418b27acd31fa30341d06609fefc410b03b5b0369912f94f323f I0216 18:14:33.197733 1 recorder.go:75] Recording config/pod/openshift-dns/logs/dns-default-jg7sl/kube-rbac-proxy_current.log with fingerprint=5de35055280444bb6c0546a40707a66f2d5e32fa91e146772b208dd5bef385e7 I0216 18:14:33.197814 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-kr85l with fingerprint=4159f7fa8c2dbbe09cb4e50ca8e5d7b9456b6c0a028b6832f608e7fdebc77c7c I0216 18:14:33.197937 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-58c6774647-lmtxb with fingerprint=6cd2bea851af90cf086ca2b0556a89c2676087ea60d7eab51d4e09a0e6af1025 I0216 18:14:33.198077 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-58c6774647-vzcnt with fingerprint=4fd5223c117f9c33d055f2e934aaa5b2739a9e14445ae341c859b83b3cc09ec1 I0216 18:14:33.198091 1 recorder.go:75] Recording config/pod/openshift-image-registry/logs/image-registry-58c6774647-vzcnt/registry_current.log with fingerprint=a1d29ade02233b289518081150ca222a670cf92da2edfa838ef4363700e65275 I0216 18:14:33.198108 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-56bf9967bf-j54ts/router_current.log with fingerprint=40e7e30d737d21057e2d726da497e90b3a7b2a53573e604c7d0428b542412239 I0216 18:14:33.198127 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-56bf9967bf-s6rgl/router_current.log with fingerprint=0b78d8b21a3dfd81694e91c9e69349aeb9d12e48b665ea615e36bc43dd78be16 I0216 18:14:33.198149 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-7548757bc-hgtwt/router_current.log with fingerprint=dab71105e5b47717273374a912ee5904c014303f244f0582ef28c8fb0745f34a I0216 18:14:33.198210 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-28wrb with fingerprint=c6ecc205e589b1498e9fea66a823ca1f0d096ee8472da5172b6f2bc5dd64a770 I0216 18:14:33.198271 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-z95xg with fingerprint=7b299d9b7d86dd24c605798b1e1db006b4de14ae41c0e442c877b8a1d8199b6d I0216 18:14:33.198279 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-z95xg/serve-healthcheck-canary_current.log with fingerprint=f9339d16aa53184072428d3d75bce25d30fe1a90b778731fe0b009fc52583605 I0216 18:14:33.198285 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 7.050039446s to process 27 records I0216 18:14:38.776819 1 tasks_processing.go:74] worker 5 stopped. I0216 18:14:38.776861 1 recorder.go:75] Recording config/installplans with fingerprint=b6ae0e2549358513c087729c711e8e1ad6f2144adc0ffa716b1a475ed1e6ddde I0216 18:14:38.776873 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.63239123s to process 1 records I0216 18:14:39.576437 1 tasks_processing.go:74] worker 22 stopped. I0216 18:14:39.576758 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=1311872d7f999a9a3cb4c9de67f16a5b436bcd458de4ff4511dfb4600acbb28c I0216 18:14:39.576779 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.40649675s to process 1 records E0216 18:14:39.576861 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.444s with: function \"machines\" failed with an error, function \"active_alerts\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"silenced_alerts\" failed with an error, function \"support_secret\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error, unable to record function \"operators_pods_and_events\" record \"config/pod/openshift-console-operator/console-operator-575cd97545-smtwj.json\"" I0216 18:14:39.577970 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machines" failed with an error, function "active_alerts" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "tsdb_status" failed with an error, function "metrics" failed with an error, function "silenced_alerts" failed with an error, function "support_secret" failed with an error, function "machine_healthchecks" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error, unable to record function "operators_pods_and_events" record "config/pod/openshift-console-operator/console-operator-575cd97545-smtwj.json" I0216 18:14:39.577984 1 periodic.go:209] Running workloads gatherer I0216 18:14:39.578001 1 tasks_processing.go:45] number of workers: 2 I0216 18:14:39.578009 1 tasks_processing.go:69] worker 1 listening for tasks. I0216 18:14:39.578014 1 tasks_processing.go:71] worker 1 working on workload_info task. I0216 18:14:39.578023 1 tasks_processing.go:69] worker 0 listening for tasks. I0216 18:14:39.578037 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0216 18:14:39.583791 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-84.ec2.internal... I0216 18:14:39.598590 1 tasks_processing.go:74] worker 0 stopped. I0216 18:14:39.598605 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 20.539263ms to process 0 records I0216 18:14:42.333464 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 3s I0216 18:14:42.354085 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 21s for image data I0216 18:14:42.363028 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (10ms) I0216 18:14:42.371327 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (8ms) I0216 18:14:42.379506 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (8ms) I0216 18:14:42.387740 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (8ms) I0216 18:14:42.398487 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (11ms) I0216 18:14:42.406704 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (8ms) I0216 18:14:42.414724 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (8ms) I0216 18:14:42.423400 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (9ms) I0216 18:14:42.432406 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (9ms) I0216 18:14:42.441620 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (9ms) I0216 18:14:42.462515 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (21ms) I0216 18:14:42.562868 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (100ms) I0216 18:14:42.663591 1 gather_workloads_info.go:387] No image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (101ms) I0216 18:14:42.763263 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (100ms) I0216 18:14:42.863528 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (100ms) I0216 18:14:42.962741 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (99ms) I0216 18:14:43.063099 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (100ms) I0216 18:14:43.163310 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (100ms) I0216 18:14:43.266108 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (103ms) I0216 18:14:43.364258 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (98ms) I0216 18:14:43.464333 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (100ms) I0216 18:14:43.563635 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (99ms) I0216 18:14:43.664687 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (101ms) I0216 18:14:43.763313 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (99ms) I0216 18:14:43.865536 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (102ms) I0216 18:14:43.963179 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (98ms) I0216 18:14:44.064063 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (101ms) I0216 18:14:44.168081 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (104ms) I0216 18:14:44.268148 1 gather_workloads_info.go:387] No image sha256:f82357030795138d2081ecc5172092222b0f4faea27e9a7a0474fbeae29111ad (100ms) I0216 18:14:44.363815 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (96ms) I0216 18:14:44.463763 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0216 18:14:44.563585 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (100ms) I0216 18:14:44.665047 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (101ms) I0216 18:14:44.764241 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (99ms) I0216 18:14:44.865010 1 gather_workloads_info.go:387] No image sha256:bffebf689df67af02a06b97d3d9bfcfdccd597350d132171ee68f3d0ed29a3f6 (101ms) I0216 18:14:44.964007 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (99ms) I0216 18:14:45.064108 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (100ms) I0216 18:14:45.164509 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0216 18:14:45.263522 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (99ms) I0216 18:14:45.263556 1 tasks_processing.go:74] worker 1 stopped. I0216 18:14:45.263801 1 recorder.go:75] Recording config/workload_info with fingerprint=a2a43801ab2cabd106285c36d41ef64a412eee096c0b66a729ca9651b9a015b4 I0216 18:14:45.263817 1 gather.go:177] gatherer "workloads" function "workload_info" took 5.685534987s to process 1 records I0216 18:14:45.263830 1 periodic.go:256] Periodic gather workloads completed in 5.685s I0216 18:14:45.263838 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0216 18:14:45.263843 1 periodic.go:209] Running conditional gatherer I0216 18:14:45.268926 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0216 18:14:45.453889 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer E0216 18:14:45.465002 1 conditional_gatherer.go:324] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host I0216 18:14:45.465090 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0216 18:14:45.470605 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0216 18:14:45.470623 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0216 18:14:45.470630 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0216 18:14:45.470633 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0216 18:14:45.470636 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0216 18:14:45.470639 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0216 18:14:45.470642 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0216 18:14:45.470645 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0216 18:14:45.470647 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0216 18:14:45.470650 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0216 18:14:45.470655 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0216 18:14:45.470689 1 tasks_processing.go:45] number of workers: 3 I0216 18:14:45.470699 1 tasks_processing.go:69] worker 2 listening for tasks. I0216 18:14:45.470705 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0216 18:14:45.470709 1 tasks_processing.go:69] worker 0 listening for tasks. I0216 18:14:45.470719 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0216 18:14:45.470721 1 tasks_processing.go:69] worker 1 listening for tasks. I0216 18:14:45.470724 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0216 18:14:45.470731 1 tasks_processing.go:74] worker 1 stopped. I0216 18:14:45.470794 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=9fafdcbf373f1f367f1d69551ea59114592cce1681189be0f3dde774e0530661 I0216 18:14:45.470808 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 941ns to process 1 records I0216 18:14:45.470841 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0216 18:14:45.470848 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.537µs to process 1 records I0216 18:14:45.470854 1 tasks_processing.go:74] worker 0 stopped. I0216 18:14:45.470999 1 tasks_processing.go:74] worker 2 stopped. I0216 18:14:45.471013 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 264.939µs to process 0 records I0216 18:14:45.471029 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0216 18:14:45.471046 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0216 18:14:45.495503 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=bfca39a64e7ecfa9ff5ea2b1c612de4c87f7edd91a27c647ad1805a037f78173 I0216 18:14:45.495649 1 diskrecorder.go:70] Writing 127 records to /var/lib/insights-operator/insights-2026-02-16-181445.tar.gz I0216 18:14:45.504775 1 diskrecorder.go:51] Wrote 127 records to disk in 9ms I0216 18:14:45.504817 1 periodic.go:278] Gathering cluster info every 2h0m0s I0216 18:14:45.504842 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0216 18:15:56.133401 1 diskrecorder.go:223] Found files to send: insights-2026-02-16-181445.tar.gz I0216 18:15:56.133434 1 insightsuploader.go:150] Checking archives to upload periodically every 15m44.101379743s I0216 18:15:56.133443 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0216 18:15:56.145409 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0216 18:15:56.385730 1 requests.go:87] Successfully reported id=2026-02-16T18:15:56Z x-rh-insights-request-id=c78c630d9642457b93a8737f9fcb4981, wrote=69249 I0216 18:15:56.385886 1 insightsuploader.go:187] Uploaded report successfully in 252.435436ms I0216 18:15:56.385925 1 controller.go:128] Initializing last reported time to 2026-02-16T18:15:56Z I0216 18:15:56.386002 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0216 18:15:56.386013 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0216 18:15:56.386022 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0216 18:15:56.391311 1 controller.go:512] The operator is healthy I0216 18:16:26.151644 1 controller.go:512] The operator is healthy I0216 18:16:57.044919 1 insightsreport.go:137] Pulling report from smart-proxy I0216 18:16:57.044945 1 insightsreport.go:149] Retrieving report I0216 18:16:57.051344 1 requests.go:110] Retrieving report for cluster: 17e28ab1-5431-4310-a2bb-bd9f61798466 I0216 18:16:57.051358 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/17e28ab1-5431-4310-a2bb-bd9f61798466/reports I0216 18:16:57.054176 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/17e28ab1-5431-4310-a2bb-bd9f61798466/reports I0216 18:16:57.502202 1 insightsreport.go:184] Report retrieved I0216 18:16:57.528412 1 insightsreport.go:239] Report retrieved correctly I0216 18:18:26.154391 1 controller.go:512] The operator is healthy I0216 18:19:26.125214 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0216 18:19:26.133233 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 18:19:26.133255 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0216 18:19:26.136927 1 secretconfigobserver.go:119] support secret does not exist I0216 18:20:26.159628 1 controller.go:512] The operator is healthy I0216 18:22:26.152138 1 controller.go:512] The operator is healthy I0216 18:24:26.121103 1 configmapobserver.go:84] configmaps "insights-config" not found I0216 18:24:26.137233 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0216 18:24:26.141457 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 18:24:26.141473 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0216 18:24:26.144915 1 secretconfigobserver.go:119] support secret does not exist I0216 18:24:26.147067 1 controller.go:512] The operator is healthy I0216 18:26:26.150386 1 controller.go:512] The operator is healthy I0216 18:28:26.150164 1 controller.go:512] The operator is healthy I0216 18:29:26.145730 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0216 18:29:26.149913 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 18:29:26.149934 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0216 18:29:26.153504 1 secretconfigobserver.go:119] support secret does not exist I0216 18:30:26.149092 1 controller.go:512] The operator is healthy I0216 18:31:45.488360 1 insightsuploader.go:146] Nothing to report since 2026-02-16T18:15:56Z I0216 18:32:26.149220 1 controller.go:512] The operator is healthy I0216 18:34:26.122087 1 configmapobserver.go:84] configmaps "insights-config" not found I0216 18:34:26.125885 1 configmapobserver.go:84] configmaps "insights-config" not found I0216 18:34:26.147959 1 controller.go:512] The operator is healthy I0216 18:34:26.153695 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0216 18:34:26.157958 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 18:34:26.157978 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0216 18:34:26.161372 1 secretconfigobserver.go:119] support secret does not exist I0216 18:36:26.151773 1 controller.go:512] The operator is healthy I0216 18:38:26.148083 1 controller.go:512] The operator is healthy I0216 18:39:26.161825 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0216 18:39:26.166101 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 18:39:26.166120 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0216 18:39:26.169517 1 secretconfigobserver.go:119] support secret does not exist I0216 18:40:26.148427 1 controller.go:512] The operator is healthy I0216 18:42:26.147398 1 controller.go:512] The operator is healthy I0216 18:44:26.122898 1 configmapobserver.go:84] configmaps "insights-config" not found I0216 18:44:26.148652 1 controller.go:512] The operator is healthy I0216 18:44:26.169750 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0216 18:44:26.174005 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 18:44:26.174022 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0216 18:44:26.177556 1 secretconfigobserver.go:119] support secret does not exist I0216 18:46:26.148149 1 controller.go:512] The operator is healthy I0216 18:47:12.664219 1 insightsuploader.go:146] Nothing to report since 2026-02-16T18:15:56Z I0216 18:48:26.148128 1 controller.go:512] The operator is healthy I0216 18:49:26.178656 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0216 18:49:26.182976 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 18:49:26.182998 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0216 18:49:26.186386 1 secretconfigobserver.go:119] support secret does not exist I0216 18:50:26.148943 1 controller.go:512] The operator is healthy I0216 18:52:26.149109 1 controller.go:512] The operator is healthy I0216 18:54:26.124386 1 configmapobserver.go:84] configmaps "insights-config" not found I0216 18:54:26.128097 1 configmapobserver.go:84] configmaps "insights-config" not found I0216 18:54:26.147208 1 controller.go:512] The operator is healthy I0216 18:54:26.187260 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0216 18:54:26.191789 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 18:54:26.191810 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0216 18:54:26.195865 1 secretconfigobserver.go:119] support secret does not exist I0216 18:56:26.148886 1 controller.go:512] The operator is healthy I0216 18:58:26.148336 1 controller.go:512] The operator is healthy I0216 18:59:26.195926 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0216 18:59:26.202824 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 18:59:26.202858 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0216 18:59:26.206322 1 secretconfigobserver.go:119] support secret does not exist I0216 19:00:26.148947 1 controller.go:512] The operator is healthy I0216 19:02:26.149301 1 controller.go:512] The operator is healthy I0216 19:03:38.796704 1 insightsuploader.go:146] Nothing to report since 2026-02-16T18:15:56Z I0216 19:04:26.125172 1 configmapobserver.go:84] configmaps "insights-config" not found I0216 19:04:26.147501 1 controller.go:512] The operator is healthy I0216 19:04:26.206821 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0216 19:04:26.211232 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0216 19:04:26.211253 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0216 19:04:26.215051 1 secretconfigobserver.go:119] support secret does not exist I0216 19:06:26.148153 1 controller.go:512] The operator is healthy