I0320 12:44:41.407970 1 cmd.go:253] Using service-serving-cert provided certificates I0320 12:44:41.408428 1 observer_polling.go:159] Starting file observer I0320 12:44:41.744426 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0320 12:44:41.744726 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0320 12:44:41.745680 1 secure_serving.go:57] Forcing use of http/1.1 only W0320 12:44:41.745705 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0320 12:44:41.745711 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0320 12:44:41.745717 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0320 12:44:41.745723 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. I0320 12:44:41.745723 1 simple_featuregate_reader.go:171] Starting feature-gate-detector W0320 12:44:41.745727 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0320 12:44:41.745761 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0320 12:44:41.749372 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0320 12:44:41.749418 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"81ba52b2-c914-4261-a7ef-d7b861149b11", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0320 12:44:41.751902 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0320 12:44:41.752124 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0320 12:44:41.752146 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0320 12:44:41.752145 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0320 12:44:41.752162 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0320 12:44:41.752161 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0320 12:44:41.752449 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0320 12:44:41.752827 1 secure_serving.go:213] Serving securely on [::]:8443 I0320 12:44:41.752863 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0320 12:44:41.755302 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0320 12:44:41.755329 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0320 12:44:41.755363 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0320 12:44:41.760305 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 12:44:41.760325 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0320 12:44:41.764263 1 secretconfigobserver.go:119] support secret does not exist I0320 12:44:41.768354 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 12:44:41.772227 1 secretconfigobserver.go:119] support secret does not exist I0320 12:44:41.775458 1 recorder.go:161] Pruning old reports every 5h27m26s, max age is 288h0m0s I0320 12:44:41.779770 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0320 12:44:41.779777 1 periodic.go:209] Running clusterconfig gatherer I0320 12:44:41.779791 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0320 12:44:41.779822 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0320 12:44:41.779830 1 insightsreport.go:296] Starting report retriever I0320 12:44:41.779835 1 tasks_processing.go:45] number of workers: 32 I0320 12:44:41.779836 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0320 12:44:41.779853 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 12:44:41.779866 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 12:44:41.779875 1 tasks_processing.go:69] worker 9 listening for tasks. I0320 12:44:41.779880 1 tasks_processing.go:71] worker 0 working on monitoring_persistent_volumes task. I0320 12:44:41.779882 1 tasks_processing.go:71] worker 9 working on operators_pods_and_events task. I0320 12:44:41.779902 1 tasks_processing.go:69] worker 21 listening for tasks. I0320 12:44:41.779912 1 tasks_processing.go:69] worker 15 listening for tasks. I0320 12:44:41.779915 1 tasks_processing.go:69] worker 19 listening for tasks. I0320 12:44:41.779920 1 tasks_processing.go:69] worker 20 listening for tasks. I0320 12:44:41.779925 1 tasks_processing.go:69] worker 18 listening for tasks. I0320 12:44:41.779929 1 tasks_processing.go:69] worker 5 listening for tasks. I0320 12:44:41.779929 1 tasks_processing.go:69] worker 10 listening for tasks. I0320 12:44:41.779934 1 tasks_processing.go:69] worker 2 listening for tasks. I0320 12:44:41.779938 1 tasks_processing.go:69] worker 11 listening for tasks. I0320 12:44:41.779943 1 tasks_processing.go:69] worker 3 listening for tasks. I0320 12:44:41.779947 1 tasks_processing.go:69] worker 12 listening for tasks. I0320 12:44:41.779947 1 tasks_processing.go:69] worker 27 listening for tasks. I0320 12:44:41.779952 1 tasks_processing.go:69] worker 4 listening for tasks. I0320 12:44:41.779954 1 tasks_processing.go:69] worker 13 listening for tasks. I0320 12:44:41.779960 1 tasks_processing.go:69] worker 17 listening for tasks. I0320 12:44:41.779961 1 tasks_processing.go:69] worker 14 listening for tasks. I0320 12:44:41.779964 1 tasks_processing.go:69] worker 16 listening for tasks. I0320 12:44:41.779968 1 tasks_processing.go:69] worker 23 listening for tasks. I0320 12:44:41.779957 1 tasks_processing.go:69] worker 22 listening for tasks. I0320 12:44:41.779976 1 tasks_processing.go:69] worker 26 listening for tasks. I0320 12:44:41.779980 1 tasks_processing.go:69] worker 6 listening for tasks. I0320 12:44:41.779978 1 tasks_processing.go:69] worker 24 listening for tasks. I0320 12:44:41.779986 1 tasks_processing.go:69] worker 8 listening for tasks. I0320 12:44:41.779971 1 tasks_processing.go:69] worker 25 listening for tasks. I0320 12:44:41.779990 1 tasks_processing.go:69] worker 31 listening for tasks. I0320 12:44:41.779971 1 tasks_processing.go:69] worker 7 listening for tasks. I0320 12:44:41.779993 1 tasks_processing.go:71] worker 1 working on aggregated_monitoring_cr_names task. I0320 12:44:41.779993 1 tasks_processing.go:69] worker 29 listening for tasks. I0320 12:44:41.779990 1 tasks_processing.go:69] worker 28 listening for tasks. I0320 12:44:41.779986 1 tasks_processing.go:69] worker 30 listening for tasks. I0320 12:44:41.780067 1 tasks_processing.go:71] worker 28 working on cluster_apiserver task. I0320 12:44:41.780072 1 tasks_processing.go:71] worker 30 working on support_secret task. I0320 12:44:41.780077 1 tasks_processing.go:71] worker 13 working on openshift_logging task. I0320 12:44:41.780136 1 tasks_processing.go:71] worker 10 working on machine_healthchecks task. I0320 12:44:41.780162 1 tasks_processing.go:71] worker 23 working on storage_classes task. I0320 12:44:41.780355 1 tasks_processing.go:71] worker 22 working on silenced_alerts task. I0320 12:44:41.780565 1 tasks_processing.go:71] worker 18 working on nodenetworkconfigurationpolicies task. I0320 12:44:41.780632 1 tasks_processing.go:71] worker 24 working on openstack_dataplanenodesets task. I0320 12:44:41.780735 1 tasks_processing.go:71] worker 5 working on pod_network_connectivity_checks task. I0320 12:44:41.780143 1 tasks_processing.go:71] worker 6 working on machine_config_pools task. I0320 12:44:41.780971 1 tasks_processing.go:71] worker 3 working on qemu_kubevirt_launcher_logs task. I0320 12:44:41.781024 1 tasks_processing.go:71] worker 2 working on feature_gates task. I0320 12:44:41.781075 1 tasks_processing.go:71] worker 21 working on install_plans task. I0320 12:44:41.781304 1 tasks_processing.go:71] worker 8 working on oauths task. I0320 12:44:41.781707 1 tasks_processing.go:71] worker 19 working on jaegers task. I0320 12:44:41.780149 1 tasks_processing.go:71] worker 17 working on dvo_metrics task. I0320 12:44:41.781917 1 tasks_processing.go:71] worker 31 working on clusterroles task. I0320 12:44:41.781725 1 tasks_processing.go:71] worker 26 working on crds task. I0320 12:44:41.782097 1 tasks_processing.go:71] worker 7 working on proxies task. I0320 12:44:41.782254 1 tasks_processing.go:71] worker 20 working on cost_management_metrics_configs task. I0320 12:44:41.782350 1 tasks_processing.go:71] worker 29 working on machine_sets task. I0320 12:44:41.780158 1 tasks_processing.go:71] worker 16 working on ceph_cluster task. I0320 12:44:41.782548 1 tasks_processing.go:71] worker 25 working on image task. I0320 12:44:41.780153 1 tasks_processing.go:71] worker 14 working on node_logs task. I0320 12:44:41.781879 1 tasks_processing.go:71] worker 11 working on ingress_certificates task. I0320 12:44:41.783489 1 tasks_processing.go:71] worker 15 working on nodes task. I0320 12:44:41.783722 1 tasks_processing.go:71] worker 12 working on container_images task. I0320 12:44:41.783796 1 tasks_processing.go:71] worker 4 working on tsdb_status task. I0320 12:44:41.783790 1 tasks_processing.go:71] worker 27 working on ingress task. I0320 12:44:41.785492 1 tasks_processing.go:71] worker 24 working on machine_configs task. I0320 12:44:41.785576 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 4.795543ms to process 0 records I0320 12:44:41.785593 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 4.867866ms to process 0 records I0320 12:44:41.785601 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 5.503448ms to process 0 records I0320 12:44:41.785614 1 tasks_processing.go:71] worker 18 working on openshift_machine_api_events task. I0320 12:44:41.785619 1 tasks_processing.go:71] worker 13 working on image_registries task. I0320 12:44:41.786996 1 tasks_processing.go:71] worker 28 working on machine_autoscalers task. E0320 12:44:41.787045 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.86.154:9091: connect: connection refused I0320 12:44:41.787004 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0320 12:44:41.787075 1 controller.go:326] The initial operator extension status is healthy I0320 12:44:41.787086 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0320 12:44:41.787092 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0320 12:44:41.787094 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0320 12:44:41.787112 1 controller.go:489] The operator is still being initialized I0320 12:44:41.787118 1 controller.go:512] The operator is healthy I0320 12:44:41.787294 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0320 12:44:41.787353 1 operator.go:288] started I0320 12:44:41.787377 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0320 12:44:41.787382 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0320 12:44:41.787298 1 recorder.go:75] Recording config/apiserver with fingerprint=f95509b2e4fe7b8992638cdc4b9bd54f471899193448b3b468f72c7494731fa2 I0320 12:44:41.787493 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 6.909425ms to process 1 records E0320 12:44:41.787521 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.86.154:9091: connect: connection refused I0320 12:44:41.787537 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 3.225865ms to process 0 records I0320 12:44:41.787550 1 tasks_processing.go:71] worker 4 working on openstack_dataplanedeployments task. I0320 12:44:41.794550 1 tasks_processing.go:71] worker 0 working on openstack_version task. I0320 12:44:41.794611 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 14.659854ms to process 0 records I0320 12:44:41.794701 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 12.398522ms to process 0 records I0320 12:44:41.794733 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 12.977333ms to process 0 records I0320 12:44:41.794813 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 12.361213ms to process 0 records E0320 12:44:41.794890 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0320 12:44:41.794933 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 14.720481ms to process 0 records I0320 12:44:41.794958 1 tasks_processing.go:71] worker 16 working on validating_webhook_configurations task. I0320 12:44:41.794951 1 tasks_processing.go:71] worker 10 working on lokistack task. I0320 12:44:41.794954 1 tasks_processing.go:71] worker 19 working on certificate_signing_requests task. I0320 12:44:41.794958 1 tasks_processing.go:71] worker 20 working on sap_pods task. E0320 12:44:41.794967 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0320 12:44:41.795694 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 14.103099ms to process 0 records I0320 12:44:41.795704 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 14.009658ms to process 0 records I0320 12:44:41.795725 1 tasks_processing.go:71] worker 6 working on metrics task. I0320 12:44:41.794974 1 tasks_processing.go:71] worker 5 working on networks task. I0320 12:44:41.795891 1 recorder.go:75] Recording config/featuregate with fingerprint=63d050101323868a9a65f3e35253c2224d84f4ab17fc6a89c230608d0f1a6689 I0320 12:44:41.795910 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 13.871115ms to process 1 records I0320 12:44:41.795921 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 12.634627ms to process 0 records E0320 12:44:41.795929 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0320 12:44:41.795938 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 15.128023ms to process 0 records I0320 12:44:41.795163 1 gather_logs.go:145] no pods in namespace were found I0320 12:44:41.796035 1 tasks_processing.go:71] worker 29 working on operators task. I0320 12:44:41.796054 1 recorder.go:75] Recording config/image with fingerprint=a46c6f7f5c54de0405169ae4adb98f2e3734f8574b1aea89c0cee673d0ddb868 I0320 12:44:41.796066 1 gather.go:177] gatherer "clusterconfig" function "image" took 12.491336ms to process 1 records I0320 12:44:41.796315 1 tasks_processing.go:71] worker 8 working on machines task. I0320 12:44:41.796357 1 tasks_processing.go:71] worker 30 working on service_accounts task. I0320 12:44:41.796363 1 tasks_processing.go:71] worker 2 working on container_runtime_configs task. I0320 12:44:41.796368 1 recorder.go:75] Recording config/oauth with fingerprint=4b4760e002ddd750ff6f97bd98e2ab9a9c472a5e2037b8476f506f2b47a427d4 I0320 12:44:41.796806 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 14.330416ms to process 1 records I0320 12:44:41.796893 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 14.99812ms to process 0 records I0320 12:44:41.796392 1 tasks_processing.go:71] worker 25 working on overlapping_namespace_uids task. I0320 12:44:41.796983 1 tasks_processing.go:71] worker 3 working on pdbs task. I0320 12:44:41.797065 1 tasks_processing.go:71] worker 27 working on version task. I0320 12:44:41.797365 1 recorder.go:75] Recording config/ingress with fingerprint=e85eea23efc68937f15fdc1a9db92bcf050794f86176b03af8fa9ff1417b78d3 I0320 12:44:41.797380 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 12.395555ms to process 1 records I0320 12:44:41.797390 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 13.741785ms to process 0 records I0320 12:44:41.797462 1 recorder.go:75] Recording config/proxy with fingerprint=840f2a0f0def068cfca1d3c736f11cf0b0f3eb7f880cfa91314ce793d111a08a I0320 12:44:41.797474 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 14.352232ms to process 1 records I0320 12:44:41.797581 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=b43736825a1c274f9f17c65600cf8c0df52f51d93dedb7a57b8352db32f9a80d I0320 12:44:41.797589 1 tasks_processing.go:71] worker 7 working on nodenetworkstates task. I0320 12:44:41.797594 1 tasks_processing.go:71] worker 23 working on config_maps task. I0320 12:44:41.797603 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=24be705da17c1ad0f6d124e464b74c59f3dbb34d036abe45711f9097cccbfed4 I0320 12:44:41.797611 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 16.543763ms to process 2 records I0320 12:44:41.797581 1 tasks_processing.go:71] worker 14 working on image_pruners task. I0320 12:44:41.797792 1 tasks_processing.go:71] worker 15 working on sap_config task. I0320 12:44:41.800434 1 recorder.go:75] Recording config/node/ip-10-0-0-106.ec2.internal with fingerprint=1273421aa2e9fb5aed2acd9af71daab17ddbe78119388bb3200118c28d8f9f86 I0320 12:44:41.800855 1 recorder.go:75] Recording config/node/ip-10-0-1-250.ec2.internal with fingerprint=332b94f3d83783908588fffca4c0542380b47718c14cab2f7a7edd221b9f48c8 I0320 12:44:41.801087 1 recorder.go:75] Recording config/node/ip-10-0-2-74.ec2.internal with fingerprint=6c675ce1ff809b6c34c2f524e1952e66fe6b5e6da09ba3ec4cd6ff259d847781 I0320 12:44:41.801143 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 13.565668ms to process 3 records I0320 12:44:41.801176 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 11.387295ms to process 0 records I0320 12:44:41.801212 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 14.151422ms to process 0 records E0320 12:44:41.801097 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.86.154:9091: connect: connection refused I0320 12:44:41.801267 1 tasks_processing.go:71] worker 6 working on schedulers task. I0320 12:44:41.801237 1 tasks_processing.go:71] worker 4 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0320 12:44:41.801245 1 tasks_processing.go:71] worker 28 working on storage_cluster task. E0320 12:44:41.801346 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.86.154:9091: connect: connection refused I0320 12:44:41.801593 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 5.531684ms to process 0 records W0320 12:44:41.803114 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 12:44:41.806755 1 tasks_processing.go:71] worker 10 working on infrastructures task. I0320 12:44:41.806788 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 11.57522ms to process 0 records I0320 12:44:41.807027 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 21.242095ms to process 0 records I0320 12:44:41.807036 1 tasks_processing.go:71] worker 18 working on mutating_webhook_configurations task. I0320 12:44:41.807402 1 tasks_processing.go:71] worker 0 working on active_alerts task. I0320 12:44:41.807659 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0320 12:44:41.807674 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0320 12:44:41.807679 1 controller.go:212] Source scaController *sca.Controller is not ready I0320 12:44:41.807683 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0320 12:44:41.807687 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0320 12:44:41.807706 1 controller.go:489] The operator is still being initialized I0320 12:44:41.807714 1 controller.go:512] The operator is healthy I0320 12:44:41.807771 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 12.837659ms to process 0 records I0320 12:44:41.808192 1 tasks_processing.go:71] worker 13 working on olm_operators task. I0320 12:44:41.808425 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=f0334445f3ad318caa64f70faec494faa56f9cfe00e0b78419eb8250cdcbfe91 I0320 12:44:41.808446 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 22.110842ms to process 1 records I0320 12:44:41.808456 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 11.563214ms to process 0 records E0320 12:44:41.808463 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0320 12:44:41.808472 1 gather.go:177] gatherer "clusterconfig" function "machines" took 11.727064ms to process 0 records I0320 12:44:41.808485 1 tasks_processing.go:71] worker 8 working on authentication task. I0320 12:44:41.808641 1 tasks_processing.go:71] worker 2 working on sap_datahubs task. I0320 12:44:41.808806 1 tasks_processing.go:71] worker 31 working on openstack_controlplanes task. I0320 12:44:41.808927 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=a3521a48db3f275027a8773e283adad6f6007aaf4e5da33ea2e615d2cb7c53d5 I0320 12:44:41.809109 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=5554be51e4ac085c27b42554b26462cfe4958c7e13e2352c68e3b38661ed96d7 I0320 12:44:41.809176 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 26.660194ms to process 2 records I0320 12:44:41.809210 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 28.71338ms to process 0 records I0320 12:44:41.809285 1 tasks_processing.go:74] worker 1 stopped. I0320 12:44:41.809452 1 tasks_processing.go:74] worker 5 stopped. I0320 12:44:41.809466 1 recorder.go:75] Recording config/network with fingerprint=cd2eabf74da5764b29da3d185a818451870394e6267b426a32104ce907ec4646 I0320 12:44:41.809587 1 gather.go:177] gatherer "clusterconfig" function "networks" took 12.963764ms to process 1 records I0320 12:44:41.809618 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 13.361596ms to process 0 records I0320 12:44:41.809684 1 tasks_processing.go:74] worker 20 stopped. I0320 12:44:41.809765 1 tasks_processing.go:74] worker 26 stopped. I0320 12:44:41.810366 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=b642bca76aa8237e01c1fd9ec96a922419ff23027a2ec56afcbe6ebc7711d376 I0320 12:44:41.810977 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=d7a20a0b739a89d4a96ff94a960d6bbdcfa08b67dcfcb90ffefc2e26b0f5fc6d I0320 12:44:41.811042 1 gather.go:177] gatherer "clusterconfig" function "crds" took 27.018836ms to process 2 records I0320 12:44:41.811075 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 11.864675ms to process 0 records I0320 12:44:41.811102 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 12.125923ms to process 0 records I0320 12:44:41.811138 1 tasks_processing.go:74] worker 7 stopped. I0320 12:44:41.811165 1 tasks_processing.go:74] worker 15 stopped. I0320 12:44:41.814852 1 tasks_processing.go:74] worker 28 stopped. I0320 12:44:41.814874 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 13.450706ms to process 0 records I0320 12:44:41.814994 1 tasks_processing.go:74] worker 19 stopped. I0320 12:44:41.815013 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 19.594793ms to process 0 records I0320 12:44:41.815870 1 tasks_processing.go:74] worker 6 stopped. I0320 12:44:41.815981 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=719bbe8c2497e7db5333c34f00157560d6f20dd69ee8c1bcf334a39f7676dbe1 I0320 12:44:41.816001 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 14.567411ms to process 1 records I0320 12:44:41.816272 1 tasks_processing.go:74] worker 3 stopped. I0320 12:44:41.816392 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=bb3b05e61032710a20ebde83af095c402d3a66bcd52ac6eef3fd4abf71731954 I0320 12:44:41.816422 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=b5c34967d173c36339f8bfadafc7db7e58a48607972727eb548c21d5c06afd2e I0320 12:44:41.816448 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=46a190b663303321228cfb015de75dced160a149a5a9f08670e2dbc78fda36e0 I0320 12:44:41.816474 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=2b393f5e49c96da693e96e94ee4c1e54ea26b99b7a487bbfe651ebac5d161f3c I0320 12:44:41.816504 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=61062bb7f80d8b890bc412459125e2941db35d57c4ddc590bc50456d20a78024 I0320 12:44:41.816545 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=8963dc497daa41231e51fe8713b74abc4eddb5fc7416147b8973774f377c461d I0320 12:44:41.816565 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=a692d048a743bda11334bea6e495f981ebf79e1db3e9bf6e0231361eeff356b4 I0320 12:44:41.816596 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=9f0e7b5ddd8b6471c37ad2cfc03ca239def8339ac6dd9e3a0f412cf4e2271c7f I0320 12:44:41.816623 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=5a6eeac29b8dd8eb712f0235dca99ad8300bfc29d3224054316eb1d63f96146c I0320 12:44:41.816652 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=ddd8ec637aaf0cbb9c3f45e9387f01ac5dd1e4fd34cc96ebc9d9b47b49b971dc I0320 12:44:41.816692 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=938e1bba5d233e0a8c3bd58a8e9e5ca404a4edeb49ff91823c9bdd6e974bf244 I0320 12:44:41.816715 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 19.235189ms to process 11 records I0320 12:44:41.816831 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=a851b2119afa22b8aa32c03052447660b69dbd8074fd11f4d4bc23e2d222e21c I0320 12:44:41.816851 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 18.863827ms to process 1 records I0320 12:44:41.816861 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 7.967414ms to process 0 records I0320 12:44:41.816871 1 tasks_processing.go:74] worker 31 stopped. I0320 12:44:41.816876 1 tasks_processing.go:74] worker 14 stopped. I0320 12:44:41.817206 1 tasks_processing.go:74] worker 25 stopped. I0320 12:44:41.817259 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0320 12:44:41.817293 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 20.285309ms to process 1 records I0320 12:44:41.817328 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 8.646018ms to process 0 records I0320 12:44:41.817347 1 tasks_processing.go:74] worker 2 stopped. I0320 12:44:41.818810 1 tasks_processing.go:74] worker 10 stopped. I0320 12:44:41.819347 1 recorder.go:75] Recording config/infrastructure with fingerprint=b7ab82a24e0caf75a3a53d9ec00607d5d3dd79cb41b43163fbae2b803ea9b456 I0320 12:44:41.819360 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 11.926476ms to process 1 records I0320 12:44:41.825401 1 tasks_processing.go:74] worker 24 stopped. I0320 12:44:41.825720 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0320 12:44:41.825781 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 39.823042ms to process 1 records I0320 12:44:41.826117 1 tasks_processing.go:74] worker 8 stopped. I0320 12:44:41.826632 1 recorder.go:75] Recording config/authentication with fingerprint=2f8f88230af6f781f322bce17730c272472d44bcd2ff56cbdf46f17742f298ea I0320 12:44:41.826688 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 17.622199ms to process 1 records I0320 12:44:41.829425 1 tasks_processing.go:74] worker 18 stopped. I0320 12:44:41.829679 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=b66ecd7bc1a418fc756040547a28fd40692b7a118c5194ebdc7a4066b70be8f0 I0320 12:44:41.829734 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=4403d6041610b16e7cb0cee33be2334ef12fb880c10672ca3fa632a542f4e87b I0320 12:44:41.829774 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=9c1e76e41a60bdf4a81f6330305021cfc40539f7bd41393c1ac98e49b2c3395b I0320 12:44:41.829786 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 22.379376ms to process 3 records I0320 12:44:41.831587 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0320 12:44:41.833596 1 tasks_processing.go:74] worker 22 stopped. I0320 12:44:41.833611 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0320 12:44:41.833629 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 53.185747ms to process 1 records I0320 12:44:41.833714 1 tasks_processing.go:74] worker 27 stopped. I0320 12:44:41.834001 1 recorder.go:75] Recording config/version with fingerprint=98b828d05940bf14fa9483215a8f591f9132ac7279bb9ae8aa3fe8a92ae19340 I0320 12:44:41.834055 1 recorder.go:75] Recording config/id with fingerprint=69dc31c8b121eaf5c98db7c58aaf6f636fb21bada5a158560252a2dd190eae54 I0320 12:44:41.834086 1 gather.go:177] gatherer "clusterconfig" function "version" took 36.543253ms to process 2 records I0320 12:44:41.834136 1 recorder.go:75] Recording config/alerts with fingerprint=13303809361515dba1e15bb9e35e645e325b3bb7534b7b5633ec859544d0cf20 I0320 12:44:41.834149 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 26.227692ms to process 1 records I0320 12:44:41.834159 1 tasks_processing.go:74] worker 0 stopped. I0320 12:44:41.834411 1 tasks_processing.go:74] worker 12 stopped. I0320 12:44:41.835699 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-4r77l with fingerprint=15d10e57e86e9f0f2a23c17682b291cd6f0b7b8ad0a8e2e0e094add4f4d2f43d I0320 12:44:41.835950 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-8jnct with fingerprint=b9a272e5e81252ca822d1826237660c6bb6a2b62940ec018cf6d6db5489372d4 I0320 12:44:41.836014 1 recorder.go:75] Recording config/running_containers with fingerprint=6513b16581d6c162b7f88329020b48a55328f8b8fbb4c2e364f2f6e8e36750ae I0320 12:44:41.836023 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 50.602026ms to process 3 records I0320 12:44:41.840313 1 tasks_processing.go:74] worker 16 stopped. I0320 12:44:41.840589 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=031b8e3ae0110bfb89999f8dadd22c23cf64a4d287e45867a5213f7836134757 I0320 12:44:41.840738 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=82478048e35465a22d917e3cac2cac2064767661dbf147c40f50e3021ccdf6f3 I0320 12:44:41.840915 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=66bb2756fa5b8bf1486af75b21c10cc8750d2ad5583cb4004d6e3f3fc04bc526 I0320 12:44:41.840995 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=2fbb00521040a8f9f3272eed5b84b1a16c525cf5a6e18b2ed3b4baa0f7b9d6c9 I0320 12:44:41.841072 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=c3d8f50b68d3fa8a996d1181ff43939e3420c72addb3178bd9be9528b3049630 I0320 12:44:41.841144 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=1a188cacdb4b847de82e768f262a5805a6eeec1db145259da5abf266d8c18e5c I0320 12:44:41.841229 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=3f5c8dfb9ef0c58ba1a8746e8a51b5985748b556a5635bfd107a182d3814d621 I0320 12:44:41.841302 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=cfb90a360ceb2e8422c7f49a70cf9b4460f761105a7c1f89d73071f8c1ba35d8 I0320 12:44:41.841379 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=f0a0429c8ed8e9a7f01d0a918fac26791ee44e6a94f4a9e36230cad87993f8ff I0320 12:44:41.841467 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=c419daf14ee54f675f244be99038aebbd97abe3b25ae1578f46c20de3440dd73 I0320 12:44:41.841550 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=6c005fc4bccee2be5ad3d4d30fdc5706e6d065080c94de8e5d1a03547d491e1d I0320 12:44:41.841635 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=df310a53222b39a2796b26a9fca40a5e0e50fd50fbf1be8ceb547eff1baa252f I0320 12:44:41.841713 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=ab1925a2cd8bbe2091ec045d2eed90893661e2766964511711313ad42687a498 I0320 12:44:41.841791 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 45.265496ms to process 13 records I0320 12:44:41.849796 1 tasks_processing.go:74] worker 13 stopped. I0320 12:44:41.849869 1 recorder.go:75] Recording config/olm_operators with fingerprint=770313bfecee83ede639af4048d97ba1bd058434b19e9948fe41294ae2122670 I0320 12:44:41.849884 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 41.581121ms to process 1 records I0320 12:44:41.849896 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 48.559587ms to process 0 records I0320 12:44:41.849910 1 tasks_processing.go:74] worker 4 stopped. I0320 12:44:41.852997 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0320 12:44:41.853003 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0320 12:44:41.853087 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0320 12:44:41.855549 1 base_controller.go:82] Caches are synced for ConfigController I0320 12:44:41.855565 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0320 12:44:41.856126 1 tasks_processing.go:74] worker 23 stopped. E0320 12:44:41.856140 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0320 12:44:41.856149 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0320 12:44:41.856156 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0320 12:44:41.856169 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0320 12:44:41.856207 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0320 12:44:41.856217 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0320 12:44:41.856223 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0320 12:44:41.856235 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0320 12:44:41.856297 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0320 12:44:41.856308 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0320 12:44:41.856316 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 58.515847ms to process 7 records I0320 12:44:41.860807 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 12:44:41.887602 1 base_controller.go:82] Caches are synced for LoggingSyncer I0320 12:44:41.887617 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0320 12:44:41.888935 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0320 12:44:41.898768 1 cluster_transfer.go:102] no available accepted cluster transfer I0320 12:44:41.898791 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0320 12:44:41.908715 1 gather_cluster_operator_pods_and_events.go:121] Found 6 pods with 6 containers I0320 12:44:41.908727 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 4194304 bytes I0320 12:44:41.910641 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5754988699-jcb8f pod in namespace openshift-ingress (previous: false). I0320 12:44:41.922269 1 tasks_processing.go:74] worker 11 stopped. E0320 12:44:41.922284 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0320 12:44:41.922290 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2p5etjv5l9s11l5f27t83q5nukucgejn-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2p5etjv5l9s11l5f27t83q5nukucgejn-primary-cert-bundle-secret" not found I0320 12:44:41.922349 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=084238f49aa33835fdecbd7bd856a688dc1426a5afee650f30147989e624a489 I0320 12:44:41.922364 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 139.534393ms to process 1 records I0320 12:44:41.923081 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5754988699-lsx4w pod in namespace openshift-ingress (previous: false). I0320 12:44:41.937982 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6f94cd84c7-h5d7l pod in namespace openshift-ingress (previous: false). I0320 12:44:41.950206 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-6tlh8 pod in namespace openshift-ingress-canary (previous: false). I0320 12:44:41.961104 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-84z4p pod in namespace openshift-ingress-canary (previous: false). I0320 12:44:42.001251 1 sca.go:266] etc-pki-entitlement secret successfully created I0320 12:44:42.001267 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0320 12:44:42.020346 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-swvfk pod in namespace openshift-ingress-canary (previous: false). I0320 12:44:42.230487 1 tasks_processing.go:74] worker 9 stopped. I0320 12:44:42.230591 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=5c4c2716a13477ed94b88810f1520986f067f151cfc6b4d54c65e9244b28edf7 I0320 12:44:42.230666 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=e3acd60d73e59420ed21dd58002e2acf3cd2e8ea24d58cac945bbe949f928984 I0320 12:44:42.230706 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=565443a6fea221cfb75a641a30ad2400d3464cf0d47a0fc2cef0813de10e3d1b I0320 12:44:42.230740 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-5754988699-jcb8f/router_current.log with fingerprint=38118ad9b52d180fa91b225b9b928f52bb8e557561f52014ae713dfe368f5163 I0320 12:44:42.230782 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-5754988699-lsx4w/router_current.log with fingerprint=f8b344282351c15c5c65c9c38d08acaa42385bd087120abd6d7d666602846a9e I0320 12:44:42.230811 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-6f94cd84c7-h5d7l/router_current.log with fingerprint=f7015772b561451b39186c9b72600c14d2e18d362f8732b57da7666cde916631 I0320 12:44:42.230818 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-6tlh8/serve-healthcheck-canary_current.log with fingerprint=f0a8380185679f7a6c78cb0ac9dc7b5a043bd2d7f2cd7fde18f1eeb05088df29 I0320 12:44:42.230821 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-84z4p/serve-healthcheck-canary_current.log with fingerprint=882fc95b50a5a069867fdfcf7ca637790f05f18efab1975b20edbe7d11b06487 I0320 12:44:42.230826 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-swvfk/serve-healthcheck-canary_current.log with fingerprint=d4f5b27741c91d2f4fbb3d2fd743c664187fd1913b6023a53661976cd2afa610 I0320 12:44:42.230832 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 450.589555ms to process 9 records W0320 12:44:42.802003 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0320 12:44:43.800885 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0320 12:44:44.252411 1 tasks_processing.go:74] worker 29 stopped. I0320 12:44:44.252554 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=71667e355f8908c58fdb755f4cf05a1f9c76abdb6183140b3b3b6734316a2388 I0320 12:44:44.252610 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0320 12:44:44.252633 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=3da12ac1a4adb871a70f4a62332fbf4f64880720e8bdcc044fccf2512064e05c I0320 12:44:44.252644 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0320 12:44:44.252665 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=8d0a617ea5fa4a37778a47d85e36e6c9f183737448c5487718cb5ee00e00e9be I0320 12:44:44.252682 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0320 12:44:44.252706 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=f1cb699e353d9f473fde17245cc8d0fa620dd8fe5f36a03ada4e1ec1224e40cd I0320 12:44:44.252733 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=5ba5a8abf3b04e8332f9da84189fa3a68516b3f5e4eef09e22666a182a76c7e0 I0320 12:44:44.252757 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=8b7f55a5622da7e64c43de0a47e272c8e43e78c3199c6e4cce8dee375c8dde34 I0320 12:44:44.252789 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=106424984d050bf8ef194a2e85da1e2ba77614f4dede4d60e3e46142291adf51 I0320 12:44:44.253057 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=3adf76f961720ab59c47f2a1e795569cb38131a391367043f36223716d71782c I0320 12:44:44.253076 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=d3b4698a311657e784533325d5ed594c34e388d7a5a50cb46e02e06f2cd7b980 I0320 12:44:44.253085 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0320 12:44:44.253101 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=571e3dd7fde06cdb55d744ff7cc4a7fd893e8b30422b7d109f95e5f96fa92929 I0320 12:44:44.253110 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0320 12:44:44.253130 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=81fe0f4ea9775530f165a999c57d362efcd9d0f1acdcda5fcc1d467a63c5d5e5 I0320 12:44:44.253140 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0320 12:44:44.253159 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=5ba6243de9b1813f4969f14576400ac59e10f5dfeca19861936be61bc849abbc I0320 12:44:44.253169 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0320 12:44:44.253189 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=f0935227ba2355805dc8c030a7b7c26247dfc9b22443eb9b7406e76109994c9d I0320 12:44:44.253291 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=8b190bbc842c8eab9d2e17e95b2e458bcdd0aee4ea61e563bf83b010e6a443c4 I0320 12:44:44.253300 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0320 12:44:44.253308 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0320 12:44:44.253338 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0320 12:44:44.253361 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=e94fdd5b30844b74df05ee97b633e6f6466bcc169a476b7cfdef59e8b1f44bdd I0320 12:44:44.253385 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=ca44630f5ac6aee65036dd5895b163b246034645ef4deb8aaffbdb9a1cc8d466 I0320 12:44:44.253394 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0320 12:44:44.253410 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=b2ee35433372f7cf74b848b3fbdbbb6f15cd67462bd9a5ca8ac037ed8550ee06 I0320 12:44:44.253419 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0320 12:44:44.253437 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=c205ab0cfb2330acf99ee8bc8a3a7f2be4c62cb77cab8cbffbe3ddf724aab205 I0320 12:44:44.253446 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0320 12:44:44.253459 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=a943820a7722e6d510c3ffa736e59bfb66d55b826603abef46f0f901c94ef0f6 I0320 12:44:44.253474 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=565cfc3e4186bfc746bf3f1fd130b7390a63ba8974e65dd962ac83635ac13d4c I0320 12:44:44.253489 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=f033d2581c39b51c029f18e2e8a855ab7511adc41bbce52778f338bb9b8d99e1 I0320 12:44:44.253525 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=06f894678fc402014f578711cf813ec3847aa03b8e9978a166d1a9f2230c6f0d I0320 12:44:44.253536 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0320 12:44:44.253560 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=ba1cf0cc271f7cd0c8cdf7d5e6c845936e3d279b5b644dbb37af33f303fdad8d I0320 12:44:44.253575 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0320 12:44:44.253607 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0320 12:44:44.253618 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.456351339s to process 39 records W0320 12:44:44.801374 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0320 12:44:45.801246 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0320 12:44:46.801208 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0320 12:44:46.801234 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0320 12:44:46.801248 1 tasks_processing.go:74] worker 17 stopped. E0320 12:44:46.801257 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0320 12:44:46.801268 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0320 12:44:46.801278 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.019328046s to process 1 records I0320 12:44:54.602091 1 tasks_processing.go:74] worker 21 stopped. I0320 12:44:54.602135 1 recorder.go:75] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0320 12:44:54.602148 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.820951388s to process 1 records I0320 12:44:55.401782 1 tasks_processing.go:74] worker 30 stopped. I0320 12:44:55.402121 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=4aff5860ec073ddfdf1bec96303c96ef7753a0d476f8315de0a2062bde3a2fd5 I0320 12:44:55.402136 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.605295114s to process 1 records E0320 12:44:55.402206 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.622s with: function \"tsdb_status\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"metrics\" failed with an error, function \"machines\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0320 12:44:55.403311 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "tsdb_status" failed with an error, function "machine_healthchecks" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "metrics" failed with an error, function "machines" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0320 12:44:55.403327 1 periodic.go:209] Running workloads gatherer I0320 12:44:55.403343 1 tasks_processing.go:45] number of workers: 2 I0320 12:44:55.403355 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 12:44:55.403362 1 tasks_processing.go:71] worker 1 working on workload_info task. I0320 12:44:55.403364 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 12:44:55.403438 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0320 12:44:55.409416 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-74.ec2.internal... I0320 12:44:55.409433 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-250.ec2.internal... I0320 12:44:55.409420 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-106.ec2.internal... I0320 12:44:55.429762 1 tasks_processing.go:74] worker 0 stopped. I0320 12:44:55.429775 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 26.309556ms to process 0 records I0320 12:44:59.704257 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 4s I0320 12:44:59.731138 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 25s for image data I0320 12:44:59.738561 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (9ms) I0320 12:44:59.746186 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (8ms) I0320 12:44:59.754438 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (8ms) I0320 12:44:59.762021 1 gather_workloads_info.go:387] No image sha256:0092c6ff139f8698c91349d507eb3182bc5d6c4ae040f672e347ff504accdea2 (8ms) I0320 12:44:59.769423 1 gather_workloads_info.go:387] No image sha256:b99613494e9d6a4a8207323872d63cc02b830fc80a8bf972e62c64381aa4cd17 (7ms) I0320 12:44:59.777441 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (8ms) I0320 12:44:59.784897 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (7ms) I0320 12:44:59.792414 1 gather_workloads_info.go:398] Found image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (8ms) I0320 12:44:59.799885 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (7ms) I0320 12:44:59.807155 1 gather_workloads_info.go:387] No image sha256:1477f3287cc75352305274f248f781c1b7c2a8696f4f7bc450c45a4a5f5b10ea (7ms) I0320 12:44:59.837893 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (31ms) I0320 12:44:59.938899 1 gather_workloads_info.go:387] No image sha256:6e5df2aab9ce759e8f957341a11c087c233307e3c8f8760d21270efc19ccd3ce (101ms) I0320 12:45:00.039045 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (100ms) I0320 12:45:00.138969 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (100ms) I0320 12:45:00.240593 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (102ms) I0320 12:45:00.339709 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (99ms) I0320 12:45:00.440475 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (101ms) I0320 12:45:00.540402 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (100ms) I0320 12:45:00.641114 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (101ms) I0320 12:45:00.739189 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (98ms) I0320 12:45:00.841168 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (102ms) I0320 12:45:00.939617 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (98ms) I0320 12:45:01.039817 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0320 12:45:01.139110 1 gather_workloads_info.go:387] No image sha256:101993f6e65d6fda8b90c583b718b720809182961f69bc777295f47e6df4d7ce (99ms) I0320 12:45:01.240048 1 gather_workloads_info.go:398] Found image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (101ms) I0320 12:45:01.339959 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (100ms) I0320 12:45:01.439762 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (100ms) I0320 12:45:01.539049 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (99ms) I0320 12:45:01.639814 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (101ms) I0320 12:45:01.738705 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (99ms) I0320 12:45:01.839353 1 gather_workloads_info.go:387] No image sha256:90bdf9e678fe18e35401e34e1cb0912d822e9cb32b9e41cc545870540e9a87f2 (101ms) I0320 12:45:01.939653 1 gather_workloads_info.go:387] No image sha256:513b1166e376feca472fdd5174f6abf0abf02925d6a21b7b4d89a2ff2fc67bed (100ms) I0320 12:45:02.039709 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0320 12:45:02.138437 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (99ms) I0320 12:45:02.239269 1 gather_workloads_info.go:387] No image sha256:7bd3e88d6d3f878a89203df590b684c5a77bf08df4717451d3197ab74683a80e (101ms) I0320 12:45:02.339480 1 gather_workloads_info.go:387] No image sha256:0af73ad7a47c9ff53269ae2bc5163bf9e389085220c34e4bd42c92c4e7a85f6d (100ms) I0320 12:45:02.439461 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (100ms) I0320 12:45:02.539223 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (100ms) I0320 12:45:02.639368 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (100ms) I0320 12:45:02.739137 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0320 12:45:02.839175 1 gather_workloads_info.go:387] No image sha256:74713675d580a47a56b628469c4c05c41e114e0b935960d718ff0cf7e72f613d (100ms) I0320 12:45:02.939204 1 gather_workloads_info.go:387] No image sha256:1e65f9f5490fbc930d2dce96f03ddcb28990b839d7dc011ef5de5bec6c5095bf (100ms) I0320 12:45:03.039028 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (100ms) I0320 12:45:03.138972 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (100ms) I0320 12:45:03.238669 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (100ms) I0320 12:45:03.339434 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (101ms) I0320 12:45:03.439334 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0320 12:45:03.539358 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (100ms) I0320 12:45:03.539391 1 tasks_processing.go:74] worker 1 stopped. E0320 12:45:03.539402 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: received unexpected status code 500 Internal Server Error from https://10.129.0.17:8443/gather_runtime_info I0320 12:45:03.539746 1 recorder.go:75] Recording config/workload_info with fingerprint=2ecfec778828fe3723afea06a06e243f80007532de95f9020f503e9f8e990a70 I0320 12:45:03.539762 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.136021052s to process 1 records E0320 12:45:03.539788 1 periodic.go:247] "Unhandled Error" err="workloads failed after 8.136s with: function \"workload_info\" failed with an error" I0320 12:45:03.540892 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0320 12:45:03.540906 1 periodic.go:209] Running conditional gatherer I0320 12:45:03.546406 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0320 12:45:03.710337 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer E0320 12:45:03.712766 1 conditional_gatherer.go:324] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp 172.30.86.154:9091: connect: connection refused I0320 12:45:03.712838 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0320 12:45:03.718379 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0320 12:45:03.718397 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:45:03.718404 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:45:03.718409 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:45:03.718414 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:45:03.718417 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:45:03.718422 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:45:03.718427 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:45:03.718432 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:45:03.718436 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0320 12:45:03.718441 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0320 12:45:03.718462 1 tasks_processing.go:45] number of workers: 3 I0320 12:45:03.718479 1 tasks_processing.go:69] worker 2 listening for tasks. I0320 12:45:03.718486 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0320 12:45:03.718488 1 tasks_processing.go:69] worker 0 listening for tasks. I0320 12:45:03.718494 1 tasks_processing.go:69] worker 1 listening for tasks. I0320 12:45:03.718525 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0320 12:45:03.718528 1 tasks_processing.go:74] worker 1 stopped. I0320 12:45:03.718533 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0320 12:45:03.718607 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=9fafdcbf373f1f367f1d69551ea59114592cce1681189be0f3dde774e0530661 I0320 12:45:03.718625 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.342µs to process 1 records I0320 12:45:03.718665 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0320 12:45:03.718677 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 955ns to process 1 records I0320 12:45:03.718686 1 tasks_processing.go:74] worker 0 stopped. I0320 12:45:03.718802 1 tasks_processing.go:74] worker 2 stopped. I0320 12:45:03.718814 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 267.746µs to process 0 records I0320 12:45:03.718829 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0320 12:45:03.718846 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0320 12:45:03.742982 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=78c6eccd6871e0cc491a65328685aebc68d4607adb723291616417ff300967f6 I0320 12:45:03.743117 1 diskrecorder.go:70] Writing 121 records to /var/lib/insights-operator/insights-2026-03-20-124503.tar.gz I0320 12:45:03.750869 1 diskrecorder.go:51] Wrote 121 records to disk in 7ms I0320 12:45:03.750896 1 periodic.go:278] Gathering cluster info every 2h0m0s I0320 12:45:03.750910 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0320 12:46:11.780542 1 diskrecorder.go:223] Found files to send: insights-2026-03-20-124503.tar.gz I0320 12:46:11.780579 1 insightsuploader.go:150] Checking archives to upload periodically every 15m39.882196467s I0320 12:46:11.780588 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0320 12:46:11.792852 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0320 12:46:12.000157 1 requests.go:87] Successfully reported id=2026-03-20T12:46:11Z x-rh-insights-request-id=6af627ec71134c3c9196daef2a599392, wrote=57851 I0320 12:46:12.000226 1 insightsuploader.go:187] Uploaded report successfully in 219.627949ms I0320 12:46:12.000251 1 controller.go:128] Initializing last reported time to 2026-03-20T12:46:11Z I0320 12:46:12.000270 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0320 12:46:12.000279 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0320 12:46:12.000288 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0320 12:46:12.005812 1 controller.go:512] The operator is healthy I0320 12:46:41.795235 1 controller.go:512] The operator is healthy I0320 12:47:14.093625 1 insightsreport.go:137] Pulling report from smart-proxy I0320 12:47:14.093651 1 insightsreport.go:149] Retrieving report I0320 12:47:14.101912 1 requests.go:110] Retrieving report for cluster: 53bb959e-8d08-404e-905f-814a194a789a I0320 12:47:14.101925 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/53bb959e-8d08-404e-905f-814a194a789a/reports I0320 12:47:14.104664 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/53bb959e-8d08-404e-905f-814a194a789a/reports I0320 12:47:14.260451 1 insightsreport.go:184] Report retrieved I0320 12:47:14.287696 1 insightsreport.go:239] Report retrieved correctly I0320 12:48:41.793777 1 controller.go:512] The operator is healthy I0320 12:49:41.772278 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 12:49:41.779236 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 12:49:41.779255 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 12:49:41.783031 1 secretconfigobserver.go:119] support secret does not exist I0320 12:50:41.793863 1 controller.go:512] The operator is healthy I0320 12:52:41.795057 1 controller.go:512] The operator is healthy I0320 12:54:41.771259 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 12:54:41.783370 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 12:54:41.787231 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 12:54:41.787253 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 12:54:41.791179 1 secretconfigobserver.go:119] support secret does not exist I0320 12:54:41.792434 1 controller.go:512] The operator is healthy I0320 12:56:41.793567 1 controller.go:512] The operator is healthy I0320 12:58:41.793537 1 controller.go:512] The operator is healthy I0320 12:59:41.791539 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret W0320 13:00:07.341442 1 reflector.go:492] github.com/openshift/client-go/config/informers/externalversions/factory.go:125: watch of *v1.FeatureGate ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding W0320 13:00:07.341442 1 reflector.go:492] github.com/openshift/client-go/operator/informers/externalversions/factory.go:125: watch of *v1.InsightsOperator ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding W0320 13:00:07.341471 1 reflector.go:492] k8s.io/client-go/informers/factory.go:160: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding W0320 13:00:07.341534 1 reflector.go:492] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding W0320 13:00:07.341474 1 reflector.go:492] k8s.io/apiserver/pkg/authentication/request/headerrequest/requestheader_controller.go:183: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding W0320 13:00:07.341477 1 reflector.go:492] github.com/openshift/client-go/config/informers/externalversions/factory.go:125: watch of *v1.ClusterVersion ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding W0320 13:00:07.341455 1 reflector.go:492] k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209: watch of *v1.ConfigMap ended with: an error on the server ("unable to decode an event from the watch stream: http2: client connection lost") has prevented the request from succeeding I0320 13:00:08.347801 1 reflector.go:376] Caches populated for *v1.ConfigMap from k8s.io/apiserver/pkg/authentication/request/headerrequest/requestheader_controller.go:183 I0320 13:00:08.349535 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 13:00:08.349557 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 13:00:08.353612 1 secretconfigobserver.go:119] support secret does not exist I0320 13:00:08.444888 1 reflector.go:376] Caches populated for *v1.FeatureGate from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0320 13:00:08.500469 1 reflector.go:376] Caches populated for *v1.ConfigMap from k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209 I0320 13:00:08.719862 1 reflector.go:376] Caches populated for *v1.ClusterVersion from github.com/openshift/client-go/config/informers/externalversions/factory.go:125 I0320 13:00:08.763531 1 reflector.go:376] Caches populated for *v1.ConfigMap from k8s.io/client-go/informers/factory.go:160 I0320 13:00:08.765097 1 reflector.go:376] Caches populated for *v1.InsightsOperator from github.com/openshift/client-go/operator/informers/externalversions/factory.go:125 I0320 13:00:08.768092 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 13:00:08.772050 1 configmapobserver.go:84] configmaps "insights-config" not found I0320 13:00:08.896405 1 reflector.go:376] Caches populated for *v1.ConfigMap from k8s.io/apiserver/pkg/server/dynamiccertificates/configmap_cafile_content.go:209 I0320 13:00:41.795420 1 controller.go:512] The operator is healthy I0320 13:01:56.883384 1 insightsuploader.go:146] Nothing to report since 2026-03-20T12:46:11Z I0320 13:02:41.794113 1 controller.go:512] The operator is healthy I0320 13:04:41.794007 1 controller.go:512] The operator is healthy I0320 13:05:08.354282 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0320 13:05:08.361187 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0320 13:05:08.361207 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0320 13:05:08.365139 1 secretconfigobserver.go:119] support secret does not exist