I0521 19:32:23.768848 1 cmd.go:253] Using service-serving-cert provided certificates I0521 19:32:23.769277 1 observer_polling.go:159] Starting file observer I0521 19:32:24.213559 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0521 19:32:24.213749 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0521 19:32:24.214323 1 secure_serving.go:57] Forcing use of http/1.1 only W0521 19:32:24.214346 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0521 19:32:24.214352 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0521 19:32:24.214356 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0521 19:32:24.214362 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0521 19:32:24.214365 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0521 19:32:24.214367 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0521 19:32:24.214329 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0521 19:32:24.218273 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0521 19:32:24.218739 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"a27bd2ad-41b8-473c-8ab2-79be097e1353", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0521 19:32:24.219615 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0521 19:32:24.219636 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0521 19:32:24.219653 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0521 19:32:24.219655 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0521 19:32:24.219684 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:32:24.219672 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:32:24.219990 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0521 19:32:24.220324 1 secure_serving.go:213] Serving securely on [::]:8443 I0521 19:32:24.220352 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0521 19:32:24.224048 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0521 19:32:24.224068 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:32:24.224138 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0521 19:32:24.231561 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:32:24.231578 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:32:24.235688 1 secretconfigobserver.go:119] support secret does not exist I0521 19:32:24.240558 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:32:24.252394 1 recorder.go:176] Pruning old reports every 6h51m26s, max age is 288h0m0s I0521 19:32:24.255709 1 secretconfigobserver.go:119] support secret does not exist I0521 19:32:24.263876 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0521 19:32:24.263891 1 periodic.go:216] Running clusterconfig gatherer I0521 19:32:24.263893 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0521 19:32:24.263921 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0521 19:32:24.263929 1 insightsreport.go:296] Starting report retriever I0521 19:32:24.263933 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0521 19:32:24.263941 1 tasks_processing.go:45] number of workers: 64 I0521 19:32:24.263967 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:32:24.263974 1 tasks_processing.go:69] worker 3 listening for tasks. I0521 19:32:24.263967 1 tasks_processing.go:69] worker 7 listening for tasks. I0521 19:32:24.263983 1 tasks_processing.go:69] worker 9 listening for tasks. I0521 19:32:24.263976 1 tasks_processing.go:69] worker 8 listening for tasks. I0521 19:32:24.263987 1 tasks_processing.go:69] worker 5 listening for tasks. I0521 19:32:24.263992 1 tasks_processing.go:69] worker 10 listening for tasks. I0521 19:32:24.263993 1 tasks_processing.go:69] worker 6 listening for tasks. I0521 19:32:24.263995 1 tasks_processing.go:69] worker 35 listening for tasks. I0521 19:32:24.263998 1 tasks_processing.go:69] worker 4 listening for tasks. I0521 19:32:24.264000 1 tasks_processing.go:69] worker 23 listening for tasks. I0521 19:32:24.264001 1 tasks_processing.go:71] worker 6 working on ingress task. I0521 19:32:24.264004 1 tasks_processing.go:69] worker 11 listening for tasks. I0521 19:32:24.264008 1 tasks_processing.go:69] worker 25 listening for tasks. I0521 19:32:24.264008 1 tasks_processing.go:69] worker 61 listening for tasks. I0521 19:32:24.264014 1 tasks_processing.go:69] worker 62 listening for tasks. I0521 19:32:24.264013 1 tasks_processing.go:69] worker 54 listening for tasks. I0521 19:32:24.264000 1 tasks_processing.go:71] worker 10 working on sap_config task. I0521 19:32:24.264021 1 tasks_processing.go:69] worker 58 listening for tasks. I0521 19:32:24.264008 1 tasks_processing.go:69] worker 13 listening for tasks. I0521 19:32:24.264006 1 tasks_processing.go:69] worker 53 listening for tasks. I0521 19:32:24.264027 1 tasks_processing.go:69] worker 56 listening for tasks. I0521 19:32:24.264012 1 tasks_processing.go:69] worker 14 listening for tasks. I0521 19:32:24.264034 1 tasks_processing.go:69] worker 59 listening for tasks. I0521 19:32:24.264035 1 tasks_processing.go:69] worker 18 listening for tasks. I0521 19:32:24.264018 1 tasks_processing.go:69] worker 15 listening for tasks. I0521 19:32:24.263982 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:32:24.264042 1 tasks_processing.go:69] worker 19 listening for tasks. I0521 19:32:24.264019 1 tasks_processing.go:69] worker 26 listening for tasks. I0521 19:32:24.264047 1 tasks_processing.go:69] worker 63 listening for tasks. I0521 19:32:24.264048 1 tasks_processing.go:69] worker 29 listening for tasks. I0521 19:32:24.264023 1 tasks_processing.go:69] worker 16 listening for tasks. I0521 19:32:24.264037 1 tasks_processing.go:69] worker 37 listening for tasks. I0521 19:32:24.264310 1 tasks_processing.go:71] worker 2 working on clusterroles task. I0521 19:32:24.264322 1 tasks_processing.go:69] worker 20 listening for tasks. I0521 19:32:24.264220 1 tasks_processing.go:69] worker 60 listening for tasks. I0521 19:32:24.264353 1 tasks_processing.go:71] worker 60 working on service_accounts task. I0521 19:32:24.264369 1 tasks_processing.go:69] worker 33 listening for tasks. I0521 19:32:24.264380 1 tasks_processing.go:69] worker 21 listening for tasks. I0521 19:32:24.264397 1 tasks_processing.go:71] worker 21 working on machines task. I0521 19:32:24.264407 1 tasks_processing.go:69] worker 40 listening for tasks. I0521 19:32:24.264414 1 tasks_processing.go:69] worker 45 listening for tasks. I0521 19:32:24.264416 1 tasks_processing.go:69] worker 22 listening for tasks. I0521 19:32:24.264467 1 tasks_processing.go:69] worker 47 listening for tasks. I0521 19:32:24.264468 1 tasks_processing.go:69] worker 42 listening for tasks. I0521 19:32:24.264029 1 tasks_processing.go:69] worker 27 listening for tasks. I0521 19:32:24.264483 1 tasks_processing.go:69] worker 41 listening for tasks. I0521 19:32:24.264486 1 tasks_processing.go:69] worker 43 listening for tasks. I0521 19:32:24.264030 1 tasks_processing.go:69] worker 17 listening for tasks. I0521 19:32:24.264497 1 tasks_processing.go:69] worker 48 listening for tasks. I0521 19:32:24.264502 1 tasks_processing.go:71] worker 56 working on container_runtime_configs task. I0521 19:32:24.264030 1 tasks_processing.go:69] worker 36 listening for tasks. I0521 19:32:24.263987 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:32:24.264036 1 tasks_processing.go:69] worker 28 listening for tasks. I0521 19:32:24.264041 1 tasks_processing.go:69] worker 31 listening for tasks. I0521 19:32:24.264044 1 tasks_processing.go:69] worker 32 listening for tasks. I0521 19:32:24.264021 1 tasks_processing.go:69] worker 57 listening for tasks. I0521 19:32:24.264274 1 tasks_processing.go:69] worker 38 listening for tasks. I0521 19:32:24.264298 1 tasks_processing.go:69] worker 39 listening for tasks. I0521 19:32:24.264494 1 tasks_processing.go:69] worker 46 listening for tasks. I0521 19:32:24.264502 1 tasks_processing.go:71] worker 61 working on infrastructures task. I0521 19:32:24.264541 1 tasks_processing.go:71] worker 3 working on version task. I0521 19:32:24.264551 1 tasks_processing.go:69] worker 51 listening for tasks. I0521 19:32:24.264550 1 tasks_processing.go:69] worker 50 listening for tasks. I0521 19:32:24.264388 1 tasks_processing.go:71] worker 33 working on machine_healthchecks task. I0521 19:32:24.264568 1 tasks_processing.go:69] worker 44 listening for tasks. I0521 19:32:24.264567 1 tasks_processing.go:69] worker 49 listening for tasks. I0521 19:32:24.264579 1 tasks_processing.go:71] worker 7 working on machine_sets task. I0521 19:32:24.264623 1 tasks_processing.go:71] worker 49 working on ceph_cluster task. I0521 19:32:24.264635 1 tasks_processing.go:71] worker 35 working on overlapping_namespace_uids task. I0521 19:32:24.264648 1 tasks_processing.go:71] worker 23 working on mutating_webhook_configurations task. I0521 19:32:24.264655 1 tasks_processing.go:71] worker 45 working on feature_gates task. I0521 19:32:24.263970 1 tasks_processing.go:69] worker 12 listening for tasks. I0521 19:32:24.264673 1 tasks_processing.go:69] worker 30 listening for tasks. I0521 19:32:24.264708 1 tasks_processing.go:71] worker 62 working on dvo_metrics task. I0521 19:32:24.264730 1 tasks_processing.go:71] worker 14 working on node_logs task. I0521 19:32:24.264732 1 tasks_processing.go:71] worker 4 working on image_registries task. I0521 19:32:24.264793 1 tasks_processing.go:71] worker 12 working on jaegers task. I0521 19:32:24.264037 1 tasks_processing.go:69] worker 55 listening for tasks. I0521 19:32:24.264927 1 tasks_processing.go:71] worker 58 working on cluster_apiserver task. I0521 19:32:24.264020 1 tasks_processing.go:69] worker 52 listening for tasks. I0521 19:32:24.264636 1 tasks_processing.go:71] worker 40 working on openstack_controlplanes task. I0521 19:32:24.265222 1 tasks_processing.go:71] worker 54 working on active_alerts task. I0521 19:32:24.265412 1 tasks_processing.go:71] worker 59 working on tsdb_status task. I0521 19:32:24.265512 1 tasks_processing.go:71] worker 18 working on operators_pods_and_events task. I0521 19:32:24.265525 1 tasks_processing.go:71] worker 41 working on lokistack task. I0521 19:32:24.265541 1 tasks_processing.go:71] worker 37 working on ingress_certificates task. I0521 19:32:24.264003 1 tasks_processing.go:69] worker 24 listening for tasks. I0521 19:32:24.265549 1 tasks_processing.go:71] worker 43 working on storage_classes task. I0521 19:32:24.265555 1 tasks_processing.go:71] worker 20 working on nodes task. I0521 19:32:24.265563 1 tasks_processing.go:71] worker 22 working on machine_configs task. I0521 19:32:24.265567 1 tasks_processing.go:71] worker 39 working on nodenetworkconfigurationpolicies task. I0521 19:32:24.265789 1 tasks_processing.go:71] worker 27 working on sap_pods task. I0521 19:32:24.265835 1 tasks_processing.go:71] worker 55 working on pod_network_connectivity_checks task. I0521 19:32:24.265886 1 tasks_processing.go:71] worker 63 working on image_pruners task. I0521 19:32:24.265796 1 tasks_processing.go:69] worker 34 listening for tasks. I0521 19:32:24.265906 1 tasks_processing.go:71] worker 36 working on pdbs task. I0521 19:32:24.265960 1 tasks_processing.go:71] worker 0 working on container_images task. I0521 19:32:24.265271 1 tasks_processing.go:71] worker 26 working on olm_operators task. I0521 19:32:24.265886 1 tasks_processing.go:71] worker 19 working on support_secret task. I0521 19:32:24.266006 1 tasks_processing.go:71] worker 13 working on aggregated_monitoring_cr_names task. I0521 19:32:24.265541 1 tasks_processing.go:71] worker 48 working on machine_autoscalers task. I0521 19:32:24.265908 1 tasks_processing.go:71] worker 34 working on cost_management_metrics_configs task. I0521 19:32:24.265838 1 tasks_processing.go:71] worker 44 working on schedulers task. I0521 19:32:24.265576 1 tasks_processing.go:71] worker 28 working on oauths task. I0521 19:32:24.265651 1 tasks_processing.go:71] worker 46 working on networks task. I0521 19:32:24.265741 1 tasks_processing.go:71] worker 31 working on openshift_logging task. I0521 19:32:24.265751 1 tasks_processing.go:71] worker 32 working on sap_datahubs task. I0521 19:32:24.265756 1 tasks_processing.go:71] worker 8 working on openshift_machine_api_events task. I0521 19:32:24.265758 1 tasks_processing.go:71] worker 57 working on openstack_dataplanedeployments task. I0521 19:32:24.266986 1 tasks_processing.go:71] worker 9 working on node_features task. I0521 19:32:24.265765 1 tasks_processing.go:71] worker 38 working on nodenetworkstates task. I0521 19:32:24.265773 1 tasks_processing.go:71] worker 1 working on authentication task. I0521 19:32:24.265778 1 tasks_processing.go:71] worker 47 working on image task. I0521 19:32:24.265784 1 tasks_processing.go:71] worker 42 working on openstack_dataplanenodesets task. I0521 19:32:24.265894 1 tasks_processing.go:71] worker 11 working on metrics task. I0521 19:32:24.265823 1 tasks_processing.go:71] worker 25 working on config_maps task. I0521 19:32:24.265824 1 tasks_processing.go:71] worker 24 working on install_plans task. I0521 19:32:24.265828 1 tasks_processing.go:71] worker 51 working on monitoring_persistent_volumes task. I0521 19:32:24.265829 1 tasks_processing.go:71] worker 30 working on machine_config_pools task. I0521 19:32:24.265834 1 tasks_processing.go:71] worker 50 working on qemu_kubevirt_launcher_logs task. I0521 19:32:24.265864 1 tasks_processing.go:71] worker 53 working on silenced_alerts task. I0521 19:32:24.265871 1 tasks_processing.go:71] worker 15 working on storage_cluster task. I0521 19:32:24.265890 1 tasks_processing.go:71] worker 17 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0521 19:32:24.265891 1 tasks_processing.go:71] worker 16 working on certificate_signing_requests task. I0521 19:32:24.267393 1 tasks_processing.go:71] worker 52 working on openstack_version task. I0521 19:32:24.265892 1 tasks_processing.go:71] worker 5 working on proxies task. I0521 19:32:24.267401 1 tasks_processing.go:71] worker 29 working on operators task. I0521 19:32:24.268891 1 tasks_processing.go:71] worker 33 working on validating_webhook_configurations task. E0521 19:32:24.268901 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0521 19:32:24.268916 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 4.314327ms to process 0 records I0521 19:32:24.270362 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0521 19:32:24.270380 1 controller.go:368] The initial operator extension status is healthy I0521 19:32:24.270387 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:32:24.270392 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0521 19:32:24.270396 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:32:24.270411 1 controller.go:531] The operator is still being initialized I0521 19:32:24.270416 1 controller.go:554] The operator is healthy I0521 19:32:24.270496 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0521 19:32:24.270558 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0521 19:32:24.270615 1 operator.go:328] started I0521 19:32:24.270634 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer E0521 19:32:24.271778 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.130.122:9091: connect: connection refused I0521 19:32:24.271797 1 tasks_processing.go:71] worker 59 working on crds task. E0521 19:32:24.271797 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.130.122:9091: connect: connection refused E0521 19:32:24.271831 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.130.122:9091: connect: connection refused I0521 19:32:24.271856 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 6.368191ms to process 0 records E0521 19:32:24.271869 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.130.122:9091: connect: connection refused I0521 19:32:24.271891 1 tasks_processing.go:74] worker 11 stopped. I0521 19:32:24.271894 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 4.584853ms to process 0 records I0521 19:32:24.273478 1 tasks_processing.go:74] worker 7 stopped. I0521 19:32:24.273490 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 8.890761ms to process 0 records I0521 19:32:24.273500 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 8.518675ms to process 0 records I0521 19:32:24.273505 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 8.975262ms to process 0 records I0521 19:32:24.273509 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 7.089028ms to process 0 records I0521 19:32:24.273514 1 tasks_processing.go:74] worker 31 stopped. I0521 19:32:24.273519 1 tasks_processing.go:74] worker 40 stopped. I0521 19:32:24.273522 1 tasks_processing.go:74] worker 56 stopped. I0521 19:32:24.273556 1 tasks_processing.go:74] worker 21 stopped. E0521 19:32:24.273572 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0521 19:32:24.273581 1 gather.go:177] gatherer "clusterconfig" function "machines" took 9.142502ms to process 0 records I0521 19:32:24.286730 1 tasks_processing.go:74] worker 57 stopped. I0521 19:32:24.286744 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 19.843623ms to process 0 records I0521 19:32:24.286771 1 tasks_processing.go:74] worker 27 stopped. E0521 19:32:24.286772 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0521 19:32:24.286779 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 20.931308ms to process 0 records I0521 19:32:24.286794 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 22.756366ms to process 0 records I0521 19:32:24.286805 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 19.744921ms to process 0 records I0521 19:32:24.286812 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 19.7294ms to process 0 records I0521 19:32:24.286820 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 20.337574ms to process 0 records I0521 19:32:24.286827 1 tasks_processing.go:74] worker 32 stopped. I0521 19:32:24.286833 1 tasks_processing.go:74] worker 10 stopped. I0521 19:32:24.286838 1 tasks_processing.go:74] worker 9 stopped. I0521 19:32:24.286842 1 tasks_processing.go:74] worker 38 stopped. I0521 19:32:24.286926 1 tasks_processing.go:74] worker 42 stopped. I0521 19:32:24.286938 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 19.700174ms to process 0 records I0521 19:32:24.287135 1 tasks_processing.go:74] worker 12 stopped. I0521 19:32:24.287169 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 22.328845ms to process 0 records I0521 19:32:24.287180 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 22.499292ms to process 0 records I0521 19:32:24.287189 1 tasks_processing.go:74] worker 49 stopped. I0521 19:32:24.287408 1 tasks_processing.go:74] worker 58 stopped. I0521 19:32:24.287568 1 recorder.go:75] Recording config/apiserver with fingerprint=23bd2d910b48c820b4757bfddb040f803743357993ac7cae23890e18d866ab85 I0521 19:32:24.287583 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 22.456727ms to process 1 records I0521 19:32:24.287679 1 tasks_processing.go:74] worker 6 stopped. I0521 19:32:24.287717 1 recorder.go:75] Recording config/ingress with fingerprint=ea5308b9cd9e3296e6d8f239563c007f1cfcfb477070b1476921ed0b8a0bee01 I0521 19:32:24.287724 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 23.396858ms to process 1 records I0521 19:32:24.288129 1 tasks_processing.go:74] worker 45 stopped. I0521 19:32:24.288148 1 recorder.go:75] Recording config/featuregate with fingerprint=ce86f582c58dddd07852277fd119c9311a54cc1d16e1694b9cf8c0521861d695 I0521 19:32:24.288324 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 22.818805ms to process 1 records I0521 19:32:24.288536 1 tasks_processing.go:74] worker 61 stopped. I0521 19:32:24.289588 1 recorder.go:75] Recording config/infrastructure with fingerprint=e6700164b9c774f1769efb3c5f152d0260d6c4d2775b57dc15010442d421a9e5 I0521 19:32:24.289615 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 22.935627ms to process 1 records I0521 19:32:24.289661 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 20.848251ms to process 0 records I0521 19:32:24.289730 1 tasks_processing.go:74] worker 30 stopped. I0521 19:32:24.290021 1 tasks_processing.go:74] worker 15 stopped. I0521 19:32:24.290025 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 20.732759ms to process 0 records I0521 19:32:24.290042 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 22.602367ms to process 0 records I0521 19:32:24.290049 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 22.251676ms to process 0 records I0521 19:32:24.290053 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 22.213971ms to process 0 records I0521 19:32:24.290070 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0521 19:32:24.290080 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 24.177238ms to process 1 records I0521 19:32:24.290106 1 tasks_processing.go:74] worker 48 stopped. I0521 19:32:24.290114 1 tasks_processing.go:74] worker 34 stopped. I0521 19:32:24.290115 1 tasks_processing.go:74] worker 39 stopped. I0521 19:32:24.290118 1 tasks_processing.go:74] worker 35 stopped. I0521 19:32:24.290181 1 tasks_processing.go:74] worker 28 stopped. I0521 19:32:24.290280 1 recorder.go:75] Recording config/oauth with fingerprint=73dc7f7a172e872a3247af33edfb9a84f11c1f6714acd601976cee0907f9b4c5 I0521 19:32:24.290289 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 22.641289ms to process 1 records I0521 19:32:24.290363 1 tasks_processing.go:74] worker 46 stopped. I0521 19:32:24.290394 1 recorder.go:75] Recording config/network with fingerprint=1f2ac816ee400450f8ebdd512700255f5378abd828c2bbb84b8102c65f91dea7 I0521 19:32:24.290403 1 gather.go:177] gatherer "clusterconfig" function "networks" took 23.650302ms to process 1 records I0521 19:32:24.291945 1 tasks_processing.go:74] worker 63 stopped. I0521 19:32:24.292135 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=c97bf02faa52be114ba8ac365a85af4a9179c0eac6308c2e6c6d03bc818b7b80 I0521 19:32:24.292150 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 26.035404ms to process 1 records I0521 19:32:24.292237 1 tasks_processing.go:74] worker 4 stopped. I0521 19:32:24.292471 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=f2d92df3c815de5ded0c9b119925c40c56896fc7129a6beffc43844f2888ac91 I0521 19:32:24.292485 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 27.405792ms to process 1 records I0521 19:32:24.294469 1 tasks_processing.go:74] worker 19 stopped. E0521 19:32:24.294480 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0521 19:32:24.294487 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 28.462288ms to process 0 records I0521 19:32:24.295266 1 tasks_processing.go:74] worker 1 stopped. I0521 19:32:24.295429 1 recorder.go:75] Recording config/authentication with fingerprint=0c44d2a9235840193a5bb1f8175c381223321683b5dbd5104f48dec8bfd62106 I0521 19:32:24.295442 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 28.119317ms to process 1 records I0521 19:32:24.296263 1 tasks_processing.go:74] worker 20 stopped. I0521 19:32:24.296524 1 recorder.go:75] Recording config/node/ip-10-0-0-51.ec2.internal with fingerprint=2bf54528ae18bebdd1d7472c015a81e062c771f8c2413bad98650aba834b4da9 I0521 19:32:24.296578 1 recorder.go:75] Recording config/node/ip-10-0-1-114.ec2.internal with fingerprint=71c76c9128c91017e95808385c89c11f5f86ab4736f5677ed3321c17c98dcca6 I0521 19:32:24.296625 1 recorder.go:75] Recording config/node/ip-10-0-2-136.ec2.internal with fingerprint=b7b05cc431acd61c5a728879c8cabf64d0cdf245cbd08879e4875b165bea4f0a I0521 19:32:24.296633 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 30.675972ms to process 3 records I0521 19:32:24.300888 1 tasks_processing.go:74] worker 47 stopped. I0521 19:32:24.301010 1 recorder.go:75] Recording config/image with fingerprint=d6ea3268b8d6d2974bddf130f50d5c103d8a5a2d5c3a7ba6ec6f72095ca562db I0521 19:32:24.301029 1 gather.go:177] gatherer "clusterconfig" function "image" took 33.73053ms to process 1 records I0521 19:32:24.301098 1 tasks_processing.go:74] worker 43 stopped. I0521 19:32:24.301183 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=e7efccb4fb2d543e00ce1ee2fc4a0302ce09e72fa49bb9b42fdd74c848fd53e3 I0521 19:32:24.301214 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=4a32a4c53ef5861e93c56e15e5d4dcad230877fda7bffe411a25f192f2d08f5c I0521 19:32:24.301225 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 35.538838ms to process 2 records I0521 19:32:24.301259 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 34.655367ms to process 0 records I0521 19:32:24.301268 1 tasks_processing.go:74] worker 8 stopped. I0521 19:32:24.301335 1 tasks_processing.go:74] worker 55 stopped. E0521 19:32:24.301350 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0521 19:32:24.301361 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 35.441044ms to process 0 records I0521 19:32:24.301369 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 36.609645ms to process 0 records I0521 19:32:24.301375 1 tasks_processing.go:74] worker 14 stopped. I0521 19:32:24.302829 1 tasks_processing.go:74] worker 5 stopped. I0521 19:32:24.302883 1 recorder.go:75] Recording config/proxy with fingerprint=8adeb23e169b2798cb8546beddd525e875d27f0487c109b0f89237e905226a03 I0521 19:32:24.302894 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 34.878191ms to process 1 records I0521 19:32:24.304504 1 tasks_processing.go:74] worker 23 stopped. I0521 19:32:24.304654 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=535bc3d861c526c1268bf53ff7ffca1a52d2deba65c4be6d80a2c1779eeb7cd9 I0521 19:32:24.304683 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=658ecd3b4d8695dcd9f22e8f1b7f8e67043708c9a919345dbf7aea9215e81665 I0521 19:32:24.304707 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=d894565a841915d4bd1e09d31bbe8216a28c0e7e068037c4a0059dd7a2173c7e I0521 19:32:24.304714 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 39.846232ms to process 3 records W0521 19:32:24.305180 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:32:24.309083 1 tasks_processing.go:74] worker 52 stopped. I0521 19:32:24.309098 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 41.253918ms to process 0 records I0521 19:32:24.312140 1 gather_logs.go:145] no pods in namespace were found I0521 19:32:24.312159 1 tasks_processing.go:74] worker 50 stopped. I0521 19:32:24.312170 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 44.68816ms to process 0 records I0521 19:32:24.312737 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:32:24.312749 1 tasks_processing.go:74] worker 2 stopped. I0521 19:32:24.312751 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:32:24.312757 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0521 19:32:24.312763 1 controller.go:254] Source scaController *sca.Controller is not ready I0521 19:32:24.312767 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0521 19:32:24.312790 1 controller.go:531] The operator is still being initialized I0521 19:32:24.312802 1 controller.go:554] The operator is healthy I0521 19:32:24.312935 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=bcd8894252b352f9fdc5998dab0ca84ef3274992a8adc61355a12f72e6714962 I0521 19:32:24.313024 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=2770dc736c136d0eac5cff2b3ca65520350f3f0d6da638be8d93c4e5afe4a3fe I0521 19:32:24.313033 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 48.420419ms to process 2 records I0521 19:32:24.317994 1 tasks_processing.go:74] worker 41 stopped. I0521 19:32:24.318011 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 52.454519ms to process 0 records I0521 19:32:24.318482 1 tasks_processing.go:74] worker 22 stopped. E0521 19:32:24.318495 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0521 19:32:24.318544 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0521 19:32:24.318559 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 52.909582ms to process 1 records I0521 19:32:24.318692 1 tasks_processing.go:74] worker 36 stopped. I0521 19:32:24.318814 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=5b9faa3a86b78c89db2909018a660c391f549507852cd2fc1fc7e7c0bd70a319 I0521 19:32:24.318848 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=0cbe108e52bf8fc7f0ecb7ae6a982171a08b434df7fa3c30a6591a3303e84940 I0521 19:32:24.318872 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=6f3610480cb1040c719eefe544e089aea55a8ba6cc3ec541994835b71ace119b I0521 19:32:24.318895 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=d9ce6bf1894e4e7caca1455adac87c8e117ad6821fb3fb0459c445fbe0207405 I0521 19:32:24.318922 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=d69246f621af902beab1405db26b43bbaba1c926a4c544f9a13c97479268aeb5 I0521 19:32:24.318945 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=b8410a9ec7564d0866d5d13ff6ede0747ca09ce3a64c6682b0ee976b2c5dbd7b I0521 19:32:24.318979 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=bab98784c15ff7ab7c3db895d427ec57692d20525b39ddc4b0dc12ba3f69b4ea I0521 19:32:24.319005 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=fbd13c4d53e08f6ddb6f53c7b812b359b2d7e7b14ef1ea4994352cc26e7f0433 I0521 19:32:24.319028 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=9fbdc6665ba88b187445fc7bab6979883a34c1001daca6336cf4295fea56b89f I0521 19:32:24.319053 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=4417bd32a79ad3c640459131045b07c8a0c530ceddec4f6dc0fb9ffc4663176b I0521 19:32:24.319079 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=ab0b97206194822e13283d610c373d5f5200c2d3f1556db16f6f5d70121fbac7 I0521 19:32:24.319093 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 52.763981ms to process 11 records I0521 19:32:24.319155 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=b6be136822988d1b88ffe3c91f64f369005d2446916e05824039aedcbabf91bf I0521 19:32:24.319163 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 52.628228ms to process 1 records I0521 19:32:24.319171 1 tasks_processing.go:74] worker 44 stopped. I0521 19:32:24.320281 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0521 19:32:24.320293 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:32:24.320363 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:32:24.320385 1 tasks_processing.go:74] worker 18 stopped. I0521 19:32:24.320401 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 54.860932ms to process 0 records I0521 19:32:24.321470 1 tasks_processing.go:74] worker 51 stopped. I0521 19:32:24.321480 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 54.081226ms to process 0 records I0521 19:32:24.324478 1 base_controller.go:82] Caches are synced for ConfigController I0521 19:32:24.324492 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0521 19:32:24.326427 1 tasks_processing.go:74] worker 33 stopped. I0521 19:32:24.327003 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=7f5c06f7817c687b0f58740a8061f158bc7b89ea84be36af3b816b5519d96efd I0521 19:32:24.327071 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=58c77a794d7b69540dc126ff22a00341e1351ecb12eb0d78813c92065b8a2015 I0521 19:32:24.327147 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=a29da0dbb412ecf3b71ece1b23bd4d5c293e883fe5e0951b6f860ef99acd9065 I0521 19:32:24.327172 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=636b80d7eb65142fe06f7f1516de1f2fb155135c986f0360a4e9febaa48bd9bc I0521 19:32:24.327193 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=31de00626119a0620fc59c71ef68b26c57b6227df947660e2346a27b6121e740 I0521 19:32:24.327218 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=f96ec8ce0a731c9a34be52f9a3f8f5610af058d1239b89096431178c6fa365eb I0521 19:32:24.327239 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=dd364a0c28e6427e7a68be2712fad67d72c8d72d7cc570db0bafb2258b3d5ff7 I0521 19:32:24.327277 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=f8351b0fe4f62754d60820cb29c807ed3b580f06a69cc573463f7e5c2e69c373 I0521 19:32:24.327298 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=c24b0ee18bead4207211df8b1f598120a55ff1ec7e6a417bf31777b7783964aa I0521 19:32:24.327330 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=b785adbee8ddb349c6f0cf004f729e94940d90458d9e039c0e6b395c5b75979b I0521 19:32:24.327354 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=c376b1ca591193e1b33c9572aacde42d996998d79d3122b47a54d7831b9ca1d5 I0521 19:32:24.327376 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=4ad70a47497280aceeaa522061b7c9ca22735d3ed34949e02b27170ff2bc3435 I0521 19:32:24.327398 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=ee6bbf5ab9a579362195ef92bd62d7cf5ce3e59e66e47ed8f0dde4d6b504d2fa I0521 19:32:24.327407 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 57.518037ms to process 13 records I0521 19:32:24.329027 1 tasks_processing.go:74] worker 16 stopped. I0521 19:32:24.329042 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 61.329624ms to process 0 records I0521 19:32:24.337493 1 tasks_processing.go:74] worker 3 stopped. I0521 19:32:24.337768 1 recorder.go:75] Recording config/version with fingerprint=61b6251016608a2091397d30ef45372cc8ff399324ebc0c4ea1ddde856cb9371 I0521 19:32:24.337781 1 recorder.go:75] Recording config/id with fingerprint=0001f11fef4c184d35e17c620126140af35ddbc060fa9fbf4d64eb5d9fe9e13b I0521 19:32:24.337787 1 gather.go:177] gatherer "clusterconfig" function "version" took 72.938585ms to process 2 records I0521 19:32:24.338228 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:32:24.339614 1 tasks_processing.go:74] worker 26 stopped. I0521 19:32:24.339744 1 recorder.go:75] Recording config/olm_operators with fingerprint=6fce09bc5252786ae8f16998244a7f01ad9432db4c4d4aa149ccebc124c1a0f7 I0521 19:32:24.340019 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 73.625711ms to process 1 records I0521 19:32:24.341704 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0521 19:32:24.343586 1 tasks_processing.go:74] worker 59 stopped. I0521 19:32:24.344130 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=e253537e5071c76ac547bca92d1c05c2c152fdd2cd4bec984f9eaf0e3bf9a2dc I0521 19:32:24.344358 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=dcf846fe2e7f6debb89df4e77568afa725b04692f5139ffa452ce572e9e5b5f4 I0521 19:32:24.344370 1 gather.go:177] gatherer "clusterconfig" function "crds" took 71.783076ms to process 2 records I0521 19:32:24.344455 1 tasks_processing.go:74] worker 0 stopped. I0521 19:32:24.345527 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-66dbbb6468-vhkwj with fingerprint=fef34c9ffad8b4677f20d338328299e08f2abd94d0b3039e78bacbc897bbbbf8 I0521 19:32:24.345600 1 recorder.go:75] Recording config/running_containers with fingerprint=fd6db4dcd0869b0746b1deb255fe1155f81b791f68aa4eff67eaca92fa3f8538 I0521 19:32:24.345610 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 78.201401ms to process 2 records I0521 19:32:24.345620 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 78.219398ms to process 0 records I0521 19:32:24.345627 1 tasks_processing.go:74] worker 13 stopped. I0521 19:32:24.359929 1 tasks_processing.go:74] worker 17 stopped. I0521 19:32:24.359943 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 92.249607ms to process 0 records I0521 19:32:24.370695 1 base_controller.go:82] Caches are synced for LoggingSyncer I0521 19:32:24.370708 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0521 19:32:24.374907 1 tasks_processing.go:74] worker 37 stopped. E0521 19:32:24.374925 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0521 19:32:24.374934 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2qegnkaps54it628q1tq7tota2a7fesv-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2qegnkaps54it628q1tq7tota2a7fesv-primary-cert-bundle-secret" not found I0521 19:32:24.374982 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=157f490629a10d1f75ca64e33fa4f17155c16fd8f375c9a83309810a6be2633d I0521 19:32:24.374994 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 109.349926ms to process 1 records E0521 19:32:24.375005 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0521 19:32:24.375008 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0521 19:32:24.375012 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0521 19:32:24.375019 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0521 19:32:24.375064 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0521 19:32:24.375071 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0521 19:32:24.375075 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0521 19:32:24.375078 1 tasks_processing.go:74] worker 25 stopped. I0521 19:32:24.375086 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0521 19:32:24.375123 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0521 19:32:24.375130 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0521 19:32:24.375135 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 107.695906ms to process 7 records I0521 19:32:24.401753 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0521 19:32:24.406120 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found W0521 19:32:25.305217 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:32:25.327453 1 tasks_processing.go:74] worker 54 stopped. I0521 19:32:25.327489 1 recorder.go:75] Recording config/alerts with fingerprint=160b53e4fcb0433dce22f5b5b9d6ba381e4a66ad7f8f6401d99890b2b50f33ec I0521 19:32:25.327502 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.062084772s to process 1 records I0521 19:32:25.327515 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0521 19:32:25.327519 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.059929752s to process 1 records I0521 19:32:25.327525 1 tasks_processing.go:74] worker 53 stopped. W0521 19:32:26.305632 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:32:26.762273 1 tasks_processing.go:74] worker 29 stopped. I0521 19:32:26.762385 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=0f2e85361171445c1af7b7a1f836bacd70f84ccf2c567c6debb69081e546234e I0521 19:32:26.762429 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0521 19:32:26.762463 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=53c3faeb010f1786c6a6b1f7e93313100fbe4cc30a1c393552e58453f87862fa I0521 19:32:26.762473 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0521 19:32:26.762508 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=92e0b8ded5d19d27ea90731cabc1e7491cb9040ec0bcb841f637439637ca87fd I0521 19:32:26.762525 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0521 19:32:26.762545 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=dabbf268105c314a430a5b10b9cad82521a149471306340f1c4d93c2acbe6a07 I0521 19:32:26.762573 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=8f45c1a518150868078c88956353cd6c28e215147f90613f5a9cb30854ee01f4 I0521 19:32:26.762596 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=91414b47f7384014d84d6c5a9dab45edc6ef6210ef412fb31de86333ef322ba3 I0521 19:32:26.762627 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=ba73a189cff835eff8f1bc2ad0ed1fdbb1d3272f8524b32d674138336f5bea38 I0521 19:32:26.762879 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=6ee776bc76ad5d32427ef1bc892daf2a4835c3cd15866fa5b08220edf2029ac7 I0521 19:32:26.762898 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=833ace54ef1a61a4743cfb5c4ca4286704a8fa1e1b083382c1fae4c8795e5b3f I0521 19:32:26.762905 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0521 19:32:26.762921 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=e595c0b1b92e463d740304b8e01a353233f12611ee2264319831c6e78e839147 I0521 19:32:26.762931 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0521 19:32:26.762946 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=003840883567ebfb40c41d79b01d731c709cd774dea15f03e41be265eb284c8c I0521 19:32:26.762956 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0521 19:32:26.762976 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=aab869009259d607627d74c93eaf6e3551f269fcb228f2359f753ad26d5a56bd I0521 19:32:26.762986 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0521 19:32:26.763005 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=9882aaa3a3b58dc55f6c61f1f7a965bb0eba8c26c32b98c1c5dd935530fdcbac I0521 19:32:26.763106 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=fb199bc6e8c3106ac1231f8ec976b2b2096b31da3ae4a44533497d811a988fd8 I0521 19:32:26.763115 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0521 19:32:26.763122 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0521 19:32:26.763142 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0521 19:32:26.763165 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=c05540f501da3a48d7f9b400a0efb4016b54f2e53eec0b28cea8c0874eae02a3 I0521 19:32:26.763190 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=668accfa0e26652c6c846a4351b587704b50b641cdef6c555eff3ddfe8d3aad2 I0521 19:32:26.763199 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0521 19:32:26.763216 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=add252cf25574a2b026cf1718bdf3496a3be7f84c8d02c180b5724f9b908c041 I0521 19:32:26.763224 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0521 19:32:26.763256 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=c59fd60ed53a85d7b2c759817b6426583b9b8fd8e4375b5d55dd5dfc268ce029 I0521 19:32:26.763267 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0521 19:32:26.763280 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=2f70832ec126eab3c1aa3be091623088ddc933ce289660436396532e472d0995 I0521 19:32:26.763293 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=76bebe42208a5a5adcfbc2c167bc9bd3bb1fadc29d29bf533aeb60930be44975 I0521 19:32:26.763309 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=8ba54de5f860733e23d8c748e847a3b9732095e6068e0440e8f73732b44f20b6 I0521 19:32:26.763328 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=83121023cd57eb369e8829a97873172a6259263b2567df3d1e323b3a08d1c02d I0521 19:32:26.763337 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0521 19:32:26.763365 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=55189279e3327d55865ff54c276b80dd15be0d39da28a06c4f2bc0bd13ca983e I0521 19:32:26.763380 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0521 19:32:26.763386 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0521 19:32:26.763394 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.494271072s to process 39 records W0521 19:32:27.304758 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0521 19:32:28.305363 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:32:28.944638 1 controllerstatus.go:89] name=scaController healthy=false reason=Unauthorized message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: OCM API https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates returned HTTP 401: {"code":"ACCT-MGMT-15","href":"/api/accounts_mgmt/v1/errors/15","id":"15","kind":"Error","operation_id":"e3907d91-c83e-4892-bc39-d89740372dec","reason":"Unable to get payload details from JWT token: Unable to retrieve JWT token from request context"} W0521 19:32:29.303086 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:32:29.303126 1 tasks_processing.go:74] worker 62 stopped. E0521 19:32:29.303140 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:32:29.303153 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0521 19:32:29.303165 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.038399344s to process 1 records I0521 19:32:34.361594 1 controllerstatus.go:89] name=clusterTransferController healthy=false reason=Unauthorized message=failed to pull cluster transfer: OCM API https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27a6f4e65a-5428-42a8-bf6e-2dadca465f30%27+and+status+is+%27accepted%27 returned HTTP 401: {"code":"ACCT-MGMT-15","href":"/api/accounts_mgmt/v1/errors/15","id":"15","kind":"Error","operation_id":"12f2d479-e5d0-4688-a20d-6653e2070a59","reason":"Unable to get payload details from JWT token: Unable to retrieve JWT token from request context"} I0521 19:32:37.325092 1 tasks_processing.go:74] worker 24 stopped. I0521 19:32:37.325137 1 recorder.go:75] Recording config/installplans with fingerprint=95dfd5f33a9a46199239158a0ec0183b629818eca24ae41f4d706d7a3d604aec I0521 19:32:37.325149 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 13.057785458s to process 1 records I0521 19:32:38.072539 1 tasks_processing.go:74] worker 60 stopped. I0521 19:32:38.072814 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=4beec2c603fb3d8842305a07eebc66583bf101d72c28dd1b52fdf1785569c930 I0521 19:32:38.072832 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.808169234s to process 1 records E0521 19:32:38.072910 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.809s with: function \"machine_healthchecks\" failed with an error, function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_configs\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error" I0521 19:32:38.074016 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machine_healthchecks" failed with an error, function "tsdb_status" failed with an error, function "metrics" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_configs" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error I0521 19:32:38.074030 1 periodic.go:216] Running workloads gatherer I0521 19:32:38.074044 1 tasks_processing.go:45] number of workers: 2 I0521 19:32:38.074054 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:32:38.074058 1 tasks_processing.go:71] worker 1 working on workload_info task. I0521 19:32:38.074069 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:32:38.074089 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0521 19:32:38.080489 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-51.ec2.internal... I0521 19:32:38.080498 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-114.ec2.internal... I0521 19:32:38.080502 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-136.ec2.internal... I0521 19:32:38.103609 1 tasks_processing.go:74] worker 0 stopped. I0521 19:32:38.103627 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 29.501841ms to process 0 records I0521 19:32:42.654805 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 5s I0521 19:32:42.688979 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0521 19:32:42.702413 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (15ms) I0521 19:32:42.720348 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (18ms) I0521 19:32:42.742698 1 gather_workloads_info.go:387] No image sha256:3a1c7ec9bb0db343348170a70697ea3209ed81e22f1259ff694190e5d04ef250 (22ms) I0521 19:32:42.772617 1 gather_workloads_info.go:387] No image sha256:15928a5b803fa00781b408dbc4c3bfa7c5287b18584ba31f817def8a4430c725 (30ms) I0521 19:32:42.790527 1 gather_workloads_info.go:387] No image sha256:48f5559439bea030a6423652da448c25fff0c62ba293eb521843c1a8eb89ddb0 (18ms) I0521 19:32:42.809297 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (19ms) I0521 19:32:42.821650 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (12ms) I0521 19:32:42.839595 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (18ms) I0521 19:32:42.848629 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (9ms) I0521 19:32:42.857337 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (9ms) I0521 19:32:42.865360 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (8ms) I0521 19:32:42.896888 1 gather_workloads_info.go:387] No image sha256:17aefdc2f4752c23dc8f0947bb01fc34635822eeb18f2d8a5eddf8211b84d450 (32ms) I0521 19:32:42.997113 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (100ms) I0521 19:32:43.097469 1 gather_workloads_info.go:387] No image sha256:6c6d412ccf0c3d6a14645a79df6c4658c0f8725d7954cb83c3d2f0116499345a (100ms) I0521 19:32:43.198459 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (101ms) I0521 19:32:43.297359 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (99ms) I0521 19:32:43.398351 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (101ms) I0521 19:32:43.497409 1 gather_workloads_info.go:387] No image sha256:5940cc34b322325618929b1e0eb3864dae73d54c8f5b474e218dfb28123a61c7 (99ms) I0521 19:32:43.597758 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (100ms) I0521 19:32:43.696646 1 gather_workloads_info.go:387] No image sha256:5d6a2b8d2508c7da71192661e7c0dcf1335f48b6336a20189abffbe79eeff962 (99ms) I0521 19:32:43.797655 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (101ms) I0521 19:32:43.898322 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (101ms) I0521 19:32:43.997541 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (99ms) I0521 19:32:44.097947 1 gather_workloads_info.go:387] No image sha256:3380fecebaeee139c1607fbcd9346cebc92806679568e2244a8905ef0f52aad2 (100ms) I0521 19:32:44.198817 1 gather_workloads_info.go:387] No image sha256:572ebf7c7fab1a518582f3ca46c866730423cce6e8a19f614dc15fd61bf0c65a (101ms) I0521 19:32:44.300022 1 gather_workloads_info.go:398] Found image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (101ms) I0521 19:32:44.397921 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (98ms) I0521 19:32:44.497441 1 gather_workloads_info.go:387] No image sha256:0acfe677ed58031f8381cb8a4f2d8917ec4cfa95485e2a46529dffae54c75ace (99ms) I0521 19:32:44.598290 1 gather_workloads_info.go:387] No image sha256:52db05084f2a2e7d7bf4040e4865df88ec4074efbbe59a78d861be2c5ab2aa40 (101ms) I0521 19:32:44.697152 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (99ms) I0521 19:32:44.798022 1 gather_workloads_info.go:387] No image sha256:0b2e6536a0ca3afc5e64a218087a6287c74b77a359075a385f1949ca5bd51b22 (101ms) I0521 19:32:44.897554 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (100ms) I0521 19:32:44.997830 1 gather_workloads_info.go:387] No image sha256:5ff204630794311b0b37fa7b197b933ad85d76a481bf7cdb3bcbada08f0cdcbf (100ms) I0521 19:32:45.096750 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (99ms) I0521 19:32:45.196547 1 gather_workloads_info.go:387] No image sha256:53ff8efdd8f563421ebc1859ab021a3bbc1306f82d4ff5d9c21bc60ec5f57898 (100ms) I0521 19:32:45.297629 1 gather_workloads_info.go:387] No image sha256:9d6d63dce784df5863e6a0ba4c11317a26c9d756d4e62d47759c97190a71cce5 (101ms) I0521 19:32:45.396622 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (99ms) I0521 19:32:45.496751 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (100ms) I0521 19:32:45.602823 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (106ms) I0521 19:32:45.696852 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (94ms) I0521 19:32:45.797473 1 gather_workloads_info.go:387] No image sha256:bdeb1c455cffccdb6c90db238e9d371de06340b1d6105d47f66d87293f878ac1 (101ms) I0521 19:32:45.898099 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (101ms) I0521 19:32:45.997330 1 gather_workloads_info.go:387] No image sha256:9ef6dc311b880a6442b7a3a19006ddd955c7da36850bf000f5c9b68fd0862dd4 (99ms) I0521 19:32:46.097574 1 gather_workloads_info.go:387] No image sha256:1eb6a6ee3c66ac15be6bc21bb6015c595c11c15b9dfd21bd9f5ce7f7b990ebaa (100ms) I0521 19:32:46.198125 1 gather_workloads_info.go:398] Found image sha256:23204877f166455ce8f9de9b91a467eeda576f172d5f96124a68aa986c4775d6 (101ms) I0521 19:32:46.306335 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (108ms) I0521 19:32:46.397827 1 gather_workloads_info.go:387] No image sha256:15e1b5f763553e86d0230a751b0783e3b990438fd07cf96eed164bccb72ed564 (91ms) I0521 19:32:46.397861 1 tasks_processing.go:74] worker 1 stopped. I0521 19:32:46.398166 1 recorder.go:75] Recording config/workload_info with fingerprint=e5ddaa5743e4ef40c0a7ff6c2383d09fce4d24f94e9a7cf4e16cdf17594c5be5 I0521 19:32:46.398190 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.323796533s to process 1 records I0521 19:32:46.398206 1 periodic.go:263] Periodic gather workloads completed in 8.324s I0521 19:32:46.398214 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0521 19:32:46.398219 1 periodic.go:216] Running conditional gatherer I0521 19:32:46.404132 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0521 19:32:46.902806 1 conditional_gatherer.go:282] Received HTTP status code 401, trying again in 30s I0521 19:33:16.910185 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0521 19:33:32.022675 1 conditional_gatherer.go:282] Received HTTP status code 504, trying again in 1m0s I0521 19:33:32.022703 1 conditional_gatherer.go:107] received HTTP 504 Gateway Time-out from https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules. Using the default built-in configuration I0521 19:33:32.023066 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0521 19:33:33.097850 1 conditional_gatherer.go:374] alert "PrometheusKubernetesListWatchFailures" has state "pending" I0521 19:33:33.097867 1 conditional_gatherer.go:374] alert "ClusterMonitoringOperatorReconciliationErrors" has state "pending" I0521 19:33:33.097870 1 conditional_gatherer.go:374] alert "AlertmanagerReceiversNotConfigured" has state "pending" I0521 19:33:33.097873 1 conditional_gatherer.go:374] alert "KubePodNotScheduled" has state "pending" I0521 19:33:33.097876 1 conditional_gatherer.go:374] alert "KubePodNotScheduled" has state "pending" I0521 19:33:33.097878 1 conditional_gatherer.go:374] alert "Watchdog" has state "firing" I0521 19:33:33.097881 1 conditional_gatherer.go:374] alert "KubeContainerWaiting" has state "pending" I0521 19:33:33.097883 1 conditional_gatherer.go:374] alert "PrometheusNotConnectedToAlertmanagers" has state "pending" I0521 19:33:33.097885 1 conditional_gatherer.go:374] alert "TargetDown" has state "pending" I0521 19:33:33.097887 1 conditional_gatherer.go:374] alert "ThanosSidecarNoConnectionToStartedPrometheus" has state "pending" I0521 19:33:33.097964 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0521 19:33:33.105674 1 conditional_gatherer.go:392] cluster version is '4.21.9' I0521 19:33:33.105703 1 tasks_processing.go:45] number of workers: 3 I0521 19:33:33.105723 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:33:33.105723 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:33:33.105732 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0521 19:33:33.105738 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0521 19:33:33.105751 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:33:33.105769 1 tasks_processing.go:74] worker 1 stopped. I0521 19:33:33.105740 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0521 19:33:33.105854 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=5e6e9deebe7018085c2e386bec070d88cace102ade375a07031a50ddcdc08e82 I0521 19:33:33.105873 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.163µs to process 1 records I0521 19:33:33.105924 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0521 19:33:33.105935 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.037µs to process 1 records I0521 19:33:33.105943 1 tasks_processing.go:74] worker 2 stopped. I0521 19:33:33.106054 1 tasks_processing.go:74] worker 0 stopped. I0521 19:33:33.106071 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 307.733µs to process 0 records I0521 19:33:33.106097 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=HttpStatus504 message=received HTTP 504 Gateway Time-out from https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules. Using the default built-in configuration I0521 19:33:33.106115 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0521 19:33:33.137146 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=831778387db20e004b84cbde5ab03fa7db40eb7381a08bdaacb94f4946819c2b I0521 19:33:33.137315 1 diskrecorder.go:70] Writing 111 records to /var/lib/insights-operator/insights-2026-05-21-193333.tar.gz I0521 19:33:33.144011 1 diskrecorder.go:51] Wrote 111 records to disk in 6ms I0521 19:33:33.144044 1 periodic.go:285] Gathering cluster info every 2h0m0s I0521 19:33:33.144059 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0521 19:33:54.264916 1 diskrecorder.go:223] Found files to send: insights-2026-05-21-193333.tar.gz I0521 19:33:54.264947 1 insightsuploader.go:150] Checking archives to upload periodically every 15m58.880433934s I0521 19:33:54.264955 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0521 19:33:54.279468 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0521 19:34:09.365571 1 requests.go:69] gateway server https://console.redhat.com/api/ingress/v1/upload returned 401, x-rh-insights-request-id=46fc3968465f4d17add6ab94e29269a0 I0521 19:34:09.365755 1 insightsuploader.go:169] Unable to upload report after 15.1s: your Red Hat account is not enabled for remote support or your token has expired: {"errors":[{"meta":{"response_by":"gateway"},"status":401,"detail":"UHC services authentication failed"}]} I0521 19:34:09.365781 1 controllerstatus.go:80] name=insightsuploader healthy=false reason=NotAuthorized message=Reporting was not allowed: your Red Hat account is not enabled for remote support or your token has expired: {"errors":[{"meta":{"response_by":"gateway"},"status":401,"detail":"UHC services authentication failed"}]} I0521 19:34:24.278013 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:34:24.278072 1 controller.go:554] The operator is healthy I0521 19:36:24.280769 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:36:24.280808 1 controller.go:554] The operator is healthy I0521 19:37:24.256616 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 19:37:24.262411 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:37:24.262429 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 19:37:24.266685 1 secretconfigobserver.go:119] support secret does not exist I0521 19:38:24.278367 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:38:24.278404 1 controller.go:554] The operator is healthy I0521 19:40:24.277769 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:40:24.277811 1 controller.go:554] The operator is healthy I0521 19:42:24.244171 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:42:24.267193 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 19:42:24.277126 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:42:24.277143 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 19:42:24.283980 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:42:24.284012 1 controller.go:554] The operator is healthy I0521 19:42:24.290110 1 secretconfigobserver.go:119] support secret does not exist I0521 19:44:24.279405 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:44:24.279554 1 controller.go:554] The operator is healthy I0521 19:46:24.312163 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:46:24.312217 1 controller.go:554] The operator is healthy I0521 19:47:24.290950 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 19:47:24.297499 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:47:24.297519 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 19:47:24.302040 1 secretconfigobserver.go:119] support secret does not exist I0521 19:48:24.278548 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:48:24.278597 1 controller.go:554] The operator is healthy