I0521 19:32:17.281833 1 cmd.go:253] Using service-serving-cert provided certificates I0521 19:32:17.282239 1 observer_polling.go:159] Starting file observer I0521 19:32:18.017413 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0521 19:32:18.017635 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0521 19:32:18.018146 1 secure_serving.go:57] Forcing use of http/1.1 only W0521 19:32:18.018166 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0521 19:32:18.018170 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0521 19:32:18.018175 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0521 19:32:18.018178 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0521 19:32:18.018181 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0521 19:32:18.018183 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0521 19:32:18.018197 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0521 19:32:18.023935 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0521 19:32:18.023966 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"7de8c03a-8b77-413f-9213-d8a8467758c7", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0521 19:32:18.025069 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0521 19:32:18.025091 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0521 19:32:18.025086 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0521 19:32:18.025127 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:32:18.025092 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0521 19:32:18.025144 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:32:18.025485 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0521 19:32:18.025927 1 secure_serving.go:213] Serving securely on [::]:8443 I0521 19:32:18.025964 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0521 19:32:18.031487 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0521 19:32:18.031511 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:32:18.031609 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0521 19:32:18.038988 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:32:18.039021 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:32:18.045284 1 secretconfigobserver.go:119] support secret does not exist I0521 19:32:18.052224 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:32:18.059670 1 secretconfigobserver.go:119] support secret does not exist I0521 19:32:18.061824 1 recorder.go:176] Pruning old reports every 4h33m34s, max age is 288h0m0s I0521 19:32:18.068656 1 periodic.go:216] Running clusterconfig gatherer I0521 19:32:18.068663 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0521 19:32:18.068679 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0521 19:32:18.068699 1 tasks_processing.go:45] number of workers: 64 I0521 19:32:18.068701 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0521 19:32:18.068709 1 insightsreport.go:296] Starting report retriever I0521 19:32:18.068713 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0521 19:32:18.068722 1 tasks_processing.go:69] worker 10 listening for tasks. I0521 19:32:18.068728 1 tasks_processing.go:69] worker 7 listening for tasks. I0521 19:32:18.068734 1 tasks_processing.go:69] worker 8 listening for tasks. I0521 19:32:18.068739 1 tasks_processing.go:69] worker 9 listening for tasks. I0521 19:32:18.068734 1 tasks_processing.go:69] worker 6 listening for tasks. I0521 19:32:18.068746 1 tasks_processing.go:69] worker 45 listening for tasks. I0521 19:32:18.068748 1 tasks_processing.go:69] worker 46 listening for tasks. I0521 19:32:18.068755 1 tasks_processing.go:69] worker 28 listening for tasks. I0521 19:32:18.068756 1 tasks_processing.go:71] worker 45 working on infrastructures task. I0521 19:32:18.068760 1 tasks_processing.go:71] worker 28 working on machines task. I0521 19:32:18.068761 1 tasks_processing.go:69] worker 47 listening for tasks. I0521 19:32:18.068765 1 tasks_processing.go:69] worker 11 listening for tasks. I0521 19:32:18.068768 1 tasks_processing.go:69] worker 48 listening for tasks. I0521 19:32:18.068749 1 tasks_processing.go:69] worker 36 listening for tasks. I0521 19:32:18.068750 1 tasks_processing.go:71] worker 6 working on active_alerts task. I0521 19:32:18.068773 1 tasks_processing.go:69] worker 3 listening for tasks. I0521 19:32:18.068767 1 tasks_processing.go:69] worker 38 listening for tasks. I0521 19:32:18.068788 1 tasks_processing.go:69] worker 52 listening for tasks. I0521 19:32:18.068788 1 tasks_processing.go:69] worker 41 listening for tasks. I0521 19:32:18.068778 1 tasks_processing.go:69] worker 50 listening for tasks. I0521 19:32:18.068782 1 tasks_processing.go:69] worker 40 listening for tasks. I0521 19:32:18.068783 1 tasks_processing.go:69] worker 51 listening for tasks. I0521 19:32:18.068772 1 tasks_processing.go:69] worker 49 listening for tasks. I0521 19:32:18.068806 1 tasks_processing.go:69] worker 55 listening for tasks. I0521 19:32:18.068806 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:32:18.068775 1 tasks_processing.go:69] worker 39 listening for tasks. I0521 19:32:18.068795 1 tasks_processing.go:69] worker 53 listening for tasks. I0521 19:32:18.068814 1 tasks_processing.go:69] worker 42 listening for tasks. I0521 19:32:18.068817 1 tasks_processing.go:69] worker 17 listening for tasks. I0521 19:32:18.068822 1 tasks_processing.go:69] worker 58 listening for tasks. I0521 19:32:18.068796 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:32:18.068829 1 tasks_processing.go:69] worker 59 listening for tasks. I0521 19:32:18.068742 1 tasks_processing.go:71] worker 8 working on cluster_apiserver task. I0521 19:32:18.068833 1 tasks_processing.go:69] worker 18 listening for tasks. I0521 19:32:18.068838 1 tasks_processing.go:69] worker 61 listening for tasks. I0521 19:32:18.068827 1 tasks_processing.go:69] worker 16 listening for tasks. I0521 19:32:18.068842 1 tasks_processing.go:69] worker 62 listening for tasks. I0521 19:32:18.068752 1 tasks_processing.go:69] worker 27 listening for tasks. I0521 19:32:18.068848 1 tasks_processing.go:69] worker 30 listening for tasks. I0521 19:32:18.068852 1 tasks_processing.go:69] worker 21 listening for tasks. I0521 19:32:18.068846 1 tasks_processing.go:69] worker 63 listening for tasks. I0521 19:32:18.068857 1 tasks_processing.go:69] worker 22 listening for tasks. I0521 19:32:18.068858 1 tasks_processing.go:69] worker 31 listening for tasks. I0521 19:32:18.068864 1 tasks_processing.go:69] worker 34 listening for tasks. I0521 19:32:18.068864 1 tasks_processing.go:69] worker 26 listening for tasks. I0521 19:32:18.068868 1 tasks_processing.go:69] worker 24 listening for tasks. I0521 19:32:18.068872 1 tasks_processing.go:71] worker 49 working on nodenetworkstates task. I0521 19:32:18.068874 1 tasks_processing.go:69] worker 35 listening for tasks. I0521 19:32:18.068874 1 tasks_processing.go:71] worker 22 working on support_secret task. I0521 19:32:18.068877 1 tasks_processing.go:71] worker 52 working on schedulers task. I0521 19:32:18.068880 1 tasks_processing.go:71] worker 11 working on cost_management_metrics_configs task. I0521 19:32:18.068881 1 tasks_processing.go:71] worker 59 working on validating_webhook_configurations task. I0521 19:32:18.068883 1 tasks_processing.go:71] worker 35 working on image_pruners task. I0521 19:32:18.068887 1 tasks_processing.go:71] worker 40 working on overlapping_namespace_uids task. I0521 19:32:18.068882 1 tasks_processing.go:71] worker 51 working on qemu_kubevirt_launcher_logs task. I0521 19:32:18.068892 1 tasks_processing.go:71] worker 16 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0521 19:32:18.068894 1 tasks_processing.go:71] worker 55 working on clusterroles task. I0521 19:32:18.068900 1 tasks_processing.go:71] worker 39 working on version task. I0521 19:32:18.068904 1 tasks_processing.go:71] worker 48 working on ceph_cluster task. I0521 19:32:18.068756 1 tasks_processing.go:71] worker 46 working on storage_classes task. I0521 19:32:18.068869 1 tasks_processing.go:71] worker 31 working on machine_sets task. I0521 19:32:18.068834 1 tasks_processing.go:69] worker 60 listening for tasks. I0521 19:32:18.068996 1 tasks_processing.go:71] worker 60 working on dvo_metrics task. I0521 19:32:18.068858 1 tasks_processing.go:69] worker 33 listening for tasks. I0521 19:32:18.068803 1 tasks_processing.go:69] worker 12 listening for tasks. I0521 19:32:18.068809 1 tasks_processing.go:69] worker 13 listening for tasks. I0521 19:32:18.068812 1 tasks_processing.go:69] worker 56 listening for tasks. I0521 19:32:18.068816 1 tasks_processing.go:69] worker 57 listening for tasks. I0521 19:32:18.068809 1 tasks_processing.go:69] worker 5 listening for tasks. I0521 19:32:18.068825 1 tasks_processing.go:69] worker 4 listening for tasks. I0521 19:32:18.068825 1 tasks_processing.go:69] worker 44 listening for tasks. I0521 19:32:18.068825 1 tasks_processing.go:69] worker 43 listening for tasks. I0521 19:32:18.068788 1 tasks_processing.go:69] worker 19 listening for tasks. I0521 19:32:18.068834 1 tasks_processing.go:69] worker 15 listening for tasks. I0521 19:32:18.068839 1 tasks_processing.go:71] worker 10 working on sap_datahubs task. I0521 19:32:18.069207 1 tasks_processing.go:71] worker 15 working on aggregated_monitoring_cr_names task. I0521 19:32:18.069230 1 tasks_processing.go:71] worker 57 working on sap_pods task. I0521 19:32:18.069244 1 tasks_processing.go:71] worker 44 working on machine_healthchecks task. I0521 19:32:18.069246 1 tasks_processing.go:71] worker 43 working on pod_network_connectivity_checks task. I0521 19:32:18.068839 1 tasks_processing.go:69] worker 32 listening for tasks. I0521 19:32:18.069272 1 tasks_processing.go:71] worker 32 working on nodes task. I0521 19:32:18.069286 1 tasks_processing.go:71] worker 19 working on container_images task. I0521 19:32:18.069297 1 tasks_processing.go:71] worker 5 working on install_plans task. I0521 19:32:18.068743 1 tasks_processing.go:71] worker 9 working on silenced_alerts task. I0521 19:32:18.069382 1 tasks_processing.go:71] worker 12 working on feature_gates task. I0521 19:32:18.069446 1 tasks_processing.go:71] worker 33 working on crds task. I0521 19:32:18.069587 1 tasks_processing.go:71] worker 13 working on machine_autoscalers task. I0521 19:32:18.069713 1 tasks_processing.go:71] worker 56 working on openstack_dataplanedeployments task. I0521 19:32:18.069971 1 tasks_processing.go:71] worker 4 working on openstack_controlplanes task. I0521 19:32:18.068843 1 tasks_processing.go:71] worker 7 working on pdbs task. I0521 19:32:18.068846 1 tasks_processing.go:69] worker 29 listening for tasks. I0521 19:32:18.068853 1 tasks_processing.go:71] worker 36 working on openshift_machine_api_events task. I0521 19:32:18.070428 1 tasks_processing.go:71] worker 29 working on ingress_certificates task. I0521 19:32:18.068847 1 tasks_processing.go:69] worker 14 listening for tasks. I0521 19:32:18.070487 1 tasks_processing.go:71] worker 14 working on jaegers task. I0521 19:32:18.068859 1 tasks_processing.go:71] worker 21 working on openstack_dataplanenodesets task. I0521 19:32:18.068858 1 tasks_processing.go:69] worker 20 listening for tasks. I0521 19:32:18.068861 1 tasks_processing.go:69] worker 25 listening for tasks. I0521 19:32:18.068862 1 tasks_processing.go:71] worker 63 working on machine_configs task. I0521 19:32:18.070666 1 tasks_processing.go:71] worker 25 working on openstack_version task. I0521 19:32:18.070703 1 tasks_processing.go:71] worker 20 working on operators task. I0521 19:32:18.068863 1 tasks_processing.go:71] worker 3 working on lokistack task. I0521 19:32:18.068865 1 tasks_processing.go:71] worker 53 working on image_registries task. I0521 19:32:18.068866 1 tasks_processing.go:71] worker 47 working on metrics task. I0521 19:32:18.068801 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:32:18.068860 1 tasks_processing.go:69] worker 23 listening for tasks. I0521 19:32:18.068869 1 tasks_processing.go:71] worker 38 working on sap_config task. I0521 19:32:18.070964 1 tasks_processing.go:71] worker 23 working on monitoring_persistent_volumes task. I0521 19:32:18.068869 1 tasks_processing.go:71] worker 34 working on authentication task. I0521 19:32:18.068872 1 tasks_processing.go:71] worker 18 working on service_accounts task. I0521 19:32:18.068872 1 tasks_processing.go:71] worker 41 working on node_features task. I0521 19:32:18.068874 1 tasks_processing.go:71] worker 58 working on image task. I0521 19:32:18.068875 1 tasks_processing.go:71] worker 26 working on container_runtime_configs task. I0521 19:32:18.068876 1 tasks_processing.go:71] worker 0 working on node_logs task. I0521 19:32:18.068877 1 tasks_processing.go:71] worker 24 working on tsdb_status task. I0521 19:32:18.068868 1 tasks_processing.go:71] worker 42 working on storage_cluster task. I0521 19:32:18.068879 1 tasks_processing.go:71] worker 17 working on proxies task. I0521 19:32:18.068882 1 tasks_processing.go:71] worker 50 working on olm_operators task. I0521 19:32:18.068885 1 tasks_processing.go:71] worker 61 working on openshift_logging task. I0521 19:32:18.068888 1 tasks_processing.go:71] worker 62 working on certificate_signing_requests task. I0521 19:32:18.068889 1 tasks_processing.go:71] worker 2 working on config_maps task. I0521 19:32:18.068890 1 tasks_processing.go:71] worker 27 working on oauths task. I0521 19:32:18.068894 1 tasks_processing.go:71] worker 30 working on ingress task. I0521 19:32:18.068757 1 tasks_processing.go:69] worker 37 listening for tasks. I0521 19:32:18.068801 1 tasks_processing.go:69] worker 54 listening for tasks. I0521 19:32:18.070953 1 tasks_processing.go:71] worker 1 working on operators_pods_and_events task. I0521 19:32:18.074030 1 tasks_processing.go:71] worker 37 working on mutating_webhook_configurations task. I0521 19:32:18.074282 1 tasks_processing.go:71] worker 54 working on networks task. I0521 19:32:18.074296 1 tasks_processing.go:71] worker 31 working on machine_config_pools task. I0521 19:32:18.074308 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 5.332245ms to process 0 records I0521 19:32:18.074408 1 tasks_processing.go:71] worker 49 working on nodenetworkconfigurationpolicies task. I0521 19:32:18.074482 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 5.527535ms to process 0 records I0521 19:32:18.074939 1 tasks_processing.go:74] worker 10 stopped. I0521 19:32:18.074951 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 5.834643ms to process 0 records I0521 19:32:18.075313 1 tasks_processing.go:74] worker 11 stopped. I0521 19:32:18.075327 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 6.423419ms to process 0 records I0521 19:32:18.075445 1 tasks_processing.go:74] worker 48 stopped. I0521 19:32:18.075458 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 6.524168ms to process 0 records I0521 19:32:18.075472 1 tasks_processing.go:74] worker 43 stopped. E0521 19:32:18.075478 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0521 19:32:18.075487 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 6.216326ms to process 0 records I0521 19:32:18.075630 1 tasks_processing.go:74] worker 44 stopped. E0521 19:32:18.075648 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0521 19:32:18.075666 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 6.374285ms to process 0 records E0521 19:32:18.075916 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.142.176:9091: connect: connection refused I0521 19:32:18.075936 1 tasks_processing.go:74] worker 24 stopped. E0521 19:32:18.075943 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.142.176:9091: connect: connection refused I0521 19:32:18.075951 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 3.068905ms to process 0 records E0521 19:32:18.075960 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.142.176:9091: connect: connection refused I0521 19:32:18.075983 1 tasks_processing.go:74] worker 47 stopped. E0521 19:32:18.075990 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.142.176:9091: connect: connection refused I0521 19:32:18.076003 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 5.11036ms to process 0 records I0521 19:32:18.076583 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0521 19:32:18.076599 1 controller.go:368] The initial operator extension status is healthy I0521 19:32:18.076608 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:32:18.076613 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:32:18.076616 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0521 19:32:18.076630 1 controller.go:531] The operator is still being initialized I0521 19:32:18.076638 1 controller.go:554] The operator is healthy I0521 19:32:18.076703 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0521 19:32:18.076718 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0521 19:32:18.076926 1 operator.go:328] started I0521 19:32:18.076947 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0521 19:32:18.077218 1 tasks_processing.go:74] worker 35 stopped. I0521 19:32:18.077478 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=f04d8ab495016faebd261214e7cb651b29cd0a3398fb191b2a9a0626ff5e39f0 I0521 19:32:18.077492 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 8.318919ms to process 1 records I0521 19:32:18.078948 1 tasks_processing.go:74] worker 13 stopped. I0521 19:32:18.078959 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 9.340226ms to process 0 records I0521 19:32:18.080930 1 tasks_processing.go:74] worker 12 stopped. I0521 19:32:18.081047 1 recorder.go:75] Recording config/featuregate with fingerprint=d5447650a517ae6ab22562c490389a8a617511c04561bbd702890b9047e3670a I0521 19:32:18.081058 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 11.53447ms to process 1 records I0521 19:32:18.083476 1 tasks_processing.go:74] worker 56 stopped. I0521 19:32:18.083488 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 13.740764ms to process 0 records I0521 19:32:18.085198 1 tasks_processing.go:74] worker 14 stopped. I0521 19:32:18.085207 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 14.697786ms to process 0 records I0521 19:32:18.096239 1 tasks_processing.go:74] worker 4 stopped. I0521 19:32:18.096254 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 26.249285ms to process 0 records I0521 19:32:18.097137 1 tasks_processing.go:74] worker 57 stopped. I0521 19:32:18.097151 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 27.894342ms to process 0 records I0521 19:32:18.097158 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 26.371029ms to process 0 records I0521 19:32:18.097164 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 26.631307ms to process 0 records I0521 19:32:18.097169 1 tasks_processing.go:74] worker 21 stopped. I0521 19:32:18.097175 1 tasks_processing.go:74] worker 25 stopped. I0521 19:32:18.097176 1 tasks_processing.go:74] worker 3 stopped. E0521 19:32:18.097182 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0521 19:32:18.097187 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 26.493921ms to process 0 records I0521 19:32:18.097209 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 25.344681ms to process 0 records I0521 19:32:18.097219 1 tasks_processing.go:74] worker 41 stopped. E0521 19:32:18.097221 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0521 19:32:18.097228 1 tasks_processing.go:74] worker 28 stopped. I0521 19:32:18.097232 1 gather.go:177] gatherer "clusterconfig" function "machines" took 28.443484ms to process 0 records I0521 19:32:18.097295 1 tasks_processing.go:74] worker 52 stopped. I0521 19:32:18.097425 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=f205a1301f109712ebe6709e8d7a7320926acd25159c458057b54f64b302f804 I0521 19:32:18.097442 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 28.403965ms to process 1 records I0521 19:32:18.097483 1 tasks_processing.go:74] worker 26 stopped. I0521 19:32:18.097501 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 24.961822ms to process 0 records I0521 19:32:18.097510 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 26.527402ms to process 0 records I0521 19:32:18.097518 1 tasks_processing.go:74] worker 38 stopped. I0521 19:32:18.097595 1 tasks_processing.go:74] worker 42 stopped. I0521 19:32:18.097602 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 24.570133ms to process 0 records E0521 19:32:18.097609 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0521 19:32:18.097614 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 28.708097ms to process 0 records I0521 19:32:18.097618 1 tasks_processing.go:74] worker 22 stopped. I0521 19:32:18.097651 1 tasks_processing.go:74] worker 17 stopped. I0521 19:32:18.097738 1 recorder.go:75] Recording config/proxy with fingerprint=3616eee9de90cb63cb8c72734763e5626b0aa04b69f8f8ba5610c0dbd86c9636 I0521 19:32:18.097752 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 24.540507ms to process 1 records I0521 19:32:18.097897 1 tasks_processing.go:74] worker 36 stopped. I0521 19:32:18.097909 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 27.508819ms to process 0 records I0521 19:32:18.097972 1 tasks_processing.go:74] worker 58 stopped. I0521 19:32:18.098057 1 recorder.go:75] Recording config/image with fingerprint=a22c4f49bfce6055d85973f3d794b3618bedcd5d73d0659003854e9b154ec370 I0521 19:32:18.098075 1 gather.go:177] gatherer "clusterconfig" function "image" took 25.898707ms to process 1 records I0521 19:32:18.098350 1 tasks_processing.go:74] worker 61 stopped. I0521 19:32:18.098375 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 25.044936ms to process 0 records I0521 19:32:18.098401 1 tasks_processing.go:74] worker 23 stopped. I0521 19:32:18.098415 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 27.424117ms to process 0 records I0521 19:32:18.098432 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 24.004355ms to process 0 records I0521 19:32:18.098438 1 tasks_processing.go:74] worker 49 stopped. I0521 19:32:18.098498 1 tasks_processing.go:74] worker 46 stopped. I0521 19:32:18.098587 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=810630b0df430ca0b93dbf90c89140fd55e3a1623f374951084c8941f5c3a4a6 I0521 19:32:18.098614 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=fcacd50575492b82682f69cc0e066dba7903d7d7a2716fdfc724d8f9450f29db I0521 19:32:18.098625 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 29.572565ms to process 2 records I0521 19:32:18.098756 1 tasks_processing.go:74] worker 8 stopped. I0521 19:32:18.098802 1 recorder.go:75] Recording config/apiserver with fingerprint=b34148f30ae271f7c7a2d667ffa05eacda1e807235b610e31a452b3e85bfd4b6 I0521 19:32:18.098806 1 gather_logs.go:145] no pods in namespace were found I0521 19:32:18.098819 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 29.72713ms to process 1 records I0521 19:32:18.098830 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 25.374853ms to process 0 records I0521 19:32:18.098858 1 tasks_processing.go:74] worker 50 stopped. I0521 19:32:18.098940 1 tasks_processing.go:74] worker 45 stopped. I0521 19:32:18.099467 1 recorder.go:75] Recording config/infrastructure with fingerprint=ebcd0394ad0763f9c7abe80b22a793f150ff637f52b6fcdbd349e877184b1abc I0521 19:32:18.099487 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 29.926312ms to process 1 records I0521 19:32:18.099572 1 tasks_processing.go:74] worker 7 stopped. I0521 19:32:18.099575 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=2fbfbb0a051fc68605b6b7a0553a12af65db45cb60299ef9b5af2cc2ed549aba I0521 19:32:18.099618 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=cfc6afa0844ec110e8ee017514fa8432bdf86ce1836d059bde69cff1e368897b I0521 19:32:18.099636 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=37d8a4a76627050dee3783c27e98d2066169b7228a6c01ef776cb2573d250bf2 I0521 19:32:18.099649 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=79518982fc0a1185ed457292e4059d21b9bb8eb2f76396854e2b298bc13c720e I0521 19:32:18.099667 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=3ac8fcb2b6a884eebcfb2a48c150b0e226aa6bc74152e983c6cf04d36d153222 I0521 19:32:18.099680 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=a534c799b5bceb4f2d6b7ff19b7b26178601c25f132cec7bcf147086c42ab8be I0521 19:32:18.099697 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=fb3d72a12e41596576834d93d13b69e8e1348c9122cdaa3b3027340bfc5388f7 I0521 19:32:18.099713 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=74e4b486377e9dc78884775902ff80ac01844145ab159577a154893c1c7d3fe7 I0521 19:32:18.099726 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=78d192cfbff52a48a9b201bd341488050d2929ea7e0c6cbe7e6e97ea86c8f8f3 I0521 19:32:18.099740 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=6bd47d9147fe48839fb6fbe08b2a9d80274440b0d25c64baa2587eb3096a51f3 I0521 19:32:18.099756 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=c272baa5908b0789cd22f0ee1ed6397dfb3b639289bc11f16feb8cd1e8c7ab87 I0521 19:32:18.099761 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 28.512367ms to process 11 records I0521 19:32:18.099768 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 29.919542ms to process 0 records I0521 19:32:18.099804 1 tasks_processing.go:74] worker 51 stopped. I0521 19:32:18.099876 1 tasks_processing.go:74] worker 34 stopped. I0521 19:32:18.099920 1 recorder.go:75] Recording config/authentication with fingerprint=eb9a2c6152d625e3c1e2205a5765d07b93de60f99dea56ea79502b60e904c4cb I0521 19:32:18.099928 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 28.007923ms to process 1 records I0521 19:32:18.099937 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 24.918758ms to process 0 records I0521 19:32:18.099942 1 tasks_processing.go:74] worker 31 stopped. W0521 19:32:18.101392 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:32:18.102185 1 tasks_processing.go:74] worker 53 stopped. I0521 19:32:18.104243 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=2dcdcc3686ee3ba716541a90c62e5ed9f10d011106dc6c357e22496dcac7f945 I0521 19:32:18.104268 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 31.356659ms to process 1 records I0521 19:32:18.104470 1 tasks_processing.go:74] worker 59 stopped. I0521 19:32:18.104496 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=ea473fa0c73e1663088a0a6bedffd9f985dd6302340954bce1a694c77c18386e I0521 19:32:18.104577 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=2ce42717cf30008d071c2e445666f8bc91d9b0413ff390320b061b6092d0a2a3 I0521 19:32:18.104736 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=867e481e27559a022f09ac0ceefbdd6098af5e4576810d849d1e6b5d71d9fc9b I0521 19:32:18.104786 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=8fd5ac396ccf0e07a531babf8db2849ea27f553ba294ec2db9bdfae51caed32a I0521 19:32:18.104830 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=a49e1b1eaea4b9ba399a04cc8a61c285eb745003a4b4862c9e91a94f97c50908 I0521 19:32:18.104896 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=29d95f269fdd9c7c95a0b099b3cac8ffa23e14939d836dee20c06ec367f0eaff I0521 19:32:18.104954 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=0009c3a70e046e853364f64ff3a0d8f33d0c03004a3ed3b6f9cab17eb9930d56 I0521 19:32:18.105025 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=cd65fc1a74fdb9a4ea0a61413107432c09dff2352623e454c187934fa0cde65c I0521 19:32:18.105077 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=6f03be945301ad58eb2cc14d0c9c289a18c8652d9275dbc40247943582d1ee7d I0521 19:32:18.105151 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=429053b1cf4454b79ac14a5d32e9da3732cc28824e32967a756ec43064fb71c2 I0521 19:32:18.105228 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=6ffe71d2a637f79a05a478608076623fe810ff5ad2585e29ba9b522e0647ea56 I0521 19:32:18.105291 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=5127bbc73935a3001755a491778b725af1cbf6ecf0f96820bc399543972f4706 I0521 19:32:18.105348 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=1afe7075353f3db23271a03287a8ef6e41bdfde0f4ee3e899875ff87e5033f4d I0521 19:32:18.105376 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 35.356799ms to process 13 records I0521 19:32:18.105463 1 tasks_processing.go:74] worker 37 stopped. I0521 19:32:18.105530 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=20d88a2e741577ececbb1b4c37876ed831f7030ea6bc279c5e5d1a782385e8bc I0521 19:32:18.105591 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=157b0fe37950449c1dffc4c1516dbcac5e3abd38343e773c02ca1742268c7c33 I0521 19:32:18.105649 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=bc9c7a8ed6b671e2646eea480a85e0a6fc631987bf3562498f6a362a50cabac7 I0521 19:32:18.105660 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 30.795921ms to process 3 records I0521 19:32:18.105739 1 tasks_processing.go:74] worker 54 stopped. I0521 19:32:18.105862 1 recorder.go:75] Recording config/network with fingerprint=82c81b6534f7c7a6da3016e1b8b748bf85ec0a5262433f2d12d13a605398dcc7 I0521 19:32:18.105877 1 gather.go:177] gatherer "clusterconfig" function "networks" took 30.84133ms to process 1 records I0521 19:32:18.106220 1 controller.go:254] Source scaController *sca.Controller is not ready I0521 19:32:18.106234 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0521 19:32:18.106236 1 tasks_processing.go:74] worker 63 stopped. I0521 19:32:18.106239 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:32:18.106244 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:32:18.106248 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready E0521 19:32:18.106246 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0521 19:32:18.106264 1 controller.go:531] The operator is still being initialized I0521 19:32:18.106269 1 controller.go:554] The operator is healthy I0521 19:32:18.106273 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0521 19:32:18.106287 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 35.621856ms to process 1 records I0521 19:32:18.106317 1 tasks_processing.go:74] worker 40 stopped. I0521 19:32:18.106341 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0521 19:32:18.106352 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 37.420705ms to process 1 records I0521 19:32:18.106694 1 tasks_processing.go:74] worker 30 stopped. I0521 19:32:18.106871 1 recorder.go:75] Recording config/ingress with fingerprint=c8848bbac0e6de41c5139ebea1919ba287253b0de9434cdd8f48d903809c000a I0521 19:32:18.106886 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 33.150674ms to process 1 records I0521 19:32:18.106984 1 tasks_processing.go:74] worker 32 stopped. I0521 19:32:18.107851 1 recorder.go:75] Recording config/node/ip-10-0-0-131.ec2.internal with fingerprint=cc4a56de2806f969b53fe6c6aa16a4f6db926a19e7baeb14539f5df7323c5119 I0521 19:32:18.108065 1 recorder.go:75] Recording config/node/ip-10-0-1-72.ec2.internal with fingerprint=787fc6b740c0a5f69ec8105a82ef01179ee0d6caaa0efea679df080085188178 I0521 19:32:18.108241 1 recorder.go:75] Recording config/node/ip-10-0-2-237.ec2.internal with fingerprint=a5509ef0a722c2961e400193617465a1de22da3c1cf9989437654c6ec4700522 I0521 19:32:18.108291 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 37.597524ms to process 3 records I0521 19:32:18.108326 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 34.243056ms to process 0 records I0521 19:32:18.108373 1 tasks_processing.go:74] worker 0 stopped. I0521 19:32:18.108575 1 tasks_processing.go:74] worker 27 stopped. I0521 19:32:18.108878 1 recorder.go:75] Recording config/oauth with fingerprint=7bba39877b8fa4d685fd0042e799dcfba7f0fc4f2231ec45f2f0d6e139311450 I0521 19:32:18.108930 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 34.138875ms to process 1 records I0521 19:32:18.109839 1 tasks_processing.go:74] worker 39 stopped. I0521 19:32:18.110713 1 recorder.go:75] Recording config/version with fingerprint=c24a958b4a62323efae5d272b49eb97ffc82fc9c51cb8fab99dbca40ad905a6c I0521 19:32:18.110733 1 recorder.go:75] Recording config/id with fingerprint=fd293a3de59a3a7a640a9ac9317df87ecc9d38dd6a7fe3c2f21c9966ed5dfe56 I0521 19:32:18.110741 1 gather.go:177] gatherer "clusterconfig" function "version" took 40.921556ms to process 2 records I0521 19:32:18.111228 1 tasks_processing.go:74] worker 55 stopped. I0521 19:32:18.111582 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=2b0dd9fa2fa8d7aac19435413a5d0989fafb65a9ca3e96970654980ffbc3a6ee I0521 19:32:18.112248 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=50ac72ab2cf19789d9440012d3255ab359155b154eebde2d698f3f19d7ba99ce I0521 19:32:18.112268 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 42.309943ms to process 2 records I0521 19:32:18.112619 1 tasks_processing.go:74] worker 1 stopped. I0521 19:32:18.112636 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 39.054579ms to process 0 records I0521 19:32:18.113027 1 tasks_processing.go:74] worker 33 stopped. I0521 19:32:18.113684 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=2d27c3e318d06fd1f919845a26090510c30dbb8d0d495a9e5c901a88d20155dc I0521 19:32:18.113910 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=b38befaef96607ebf86156e8eac17f42400b2da2e987ba0fe2fc2333d03316b2 I0521 19:32:18.113920 1 gather.go:177] gatherer "clusterconfig" function "crds" took 43.56129ms to process 2 records I0521 19:32:18.113931 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 39.929257ms to process 0 records I0521 19:32:18.113937 1 tasks_processing.go:74] worker 62 stopped. I0521 19:32:18.117761 1 tasks_processing.go:74] worker 15 stopped. I0521 19:32:18.117780 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 48.532152ms to process 0 records I0521 19:32:18.125025 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0521 19:32:18.125131 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0521 19:32:18.125175 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:32:18.125181 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:32:18.131667 1 base_controller.go:82] Caches are synced for ConfigController I0521 19:32:18.131685 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0521 19:32:18.133347 1 tasks_processing.go:74] worker 19 stopped. I0521 19:32:18.134708 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-66dbbb6468-tb65z with fingerprint=367e104c5a9762699282bc27ddbea1410bb3778903f56c39f06f43a3eb07c03b I0521 19:32:18.134791 1 recorder.go:75] Recording config/running_containers with fingerprint=b6d60467593c6aa75747fcf8577d2915b8535d9d81db4eb026d78b52a2ca4f4d I0521 19:32:18.134801 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 64.041791ms to process 2 records E0521 19:32:18.134814 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0521 19:32:18.134819 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2qegnf9tor4r9ir2gd91uu7os8slmm6v-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2qegnf9tor4r9ir2gd91uu7os8slmm6v-primary-cert-bundle-secret" not found I0521 19:32:18.134855 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=74654263347946758f125782538c57e08cf05c65ff5954a0ffe62aa73775cbfb I0521 19:32:18.134864 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 64.034587ms to process 1 records I0521 19:32:18.134871 1 tasks_processing.go:74] worker 29 stopped. I0521 19:32:18.136828 1 tasks_processing.go:74] worker 16 stopped. I0521 19:32:18.136847 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 67.920296ms to process 0 records I0521 19:32:18.137922 1 tasks_processing.go:74] worker 2 stopped. E0521 19:32:18.137936 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0521 19:32:18.137941 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0521 19:32:18.137945 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0521 19:32:18.137954 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0521 19:32:18.137990 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0521 19:32:18.138013 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0521 19:32:18.138026 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0521 19:32:18.138039 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0521 19:32:18.138078 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0521 19:32:18.138085 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0521 19:32:18.138090 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 64.503862ms to process 7 records I0521 19:32:18.139860 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:32:18.177158 1 base_controller.go:82] Caches are synced for LoggingSyncer I0521 19:32:18.177180 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0521 19:32:18.189490 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0521 19:32:18.196145 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found W0521 19:32:19.101004 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:32:19.102600 1 tasks_processing.go:74] worker 9 stopped. I0521 19:32:19.102621 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0521 19:32:19.102633 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.033254122s to process 1 records I0521 19:32:19.102812 1 tasks_processing.go:74] worker 6 stopped. I0521 19:32:19.102843 1 recorder.go:75] Recording config/alerts with fingerprint=06dcaa4d81cd6c00a9eb4cb390a25b46982ec5e53177dc98eaf01e411aa8cc86 I0521 19:32:19.102850 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.034024305s to process 1 records W0521 19:32:20.101064 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:32:20.536459 1 tasks_processing.go:74] worker 20 stopped. I0521 19:32:20.536572 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=5fe55364bc9bc3b53f3f95f2768f14b5bd5aeae772e81058effb7569ce7159c6 I0521 19:32:20.536620 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0521 19:32:20.536655 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=986930a7c36ec2252564b2a3c119358f271ea0904fb683362eca84acd10449b2 I0521 19:32:20.536665 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0521 19:32:20.536686 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=46921abad736d49b5c109020e2f9eb708d3c2c3870859cfba79f0f043b4e1441 I0521 19:32:20.536702 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0521 19:32:20.536725 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=9fbc01937383f6df13271d122f8f7e529f0aa691cbff5cc4b9c561afebfdcaa2 I0521 19:32:20.536751 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=335f6fbab13ca662d283e21efb7a194c004616fda5de6cdaa28378243f85c376 I0521 19:32:20.536773 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=a52b969cb3ba1e8c21486451f5a686f5a17f0a5d1b2b2af21bce53d21ecff0ac I0521 19:32:20.536803 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=3524594f91e5771a01267c1a05a7a32496364f8a7a1d62d4050e643de2d18cb6 I0521 19:32:20.537086 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=9ac6cd91080f714e8860cd6fa4b06a8877dc5d22016218103d0a4414f88ba7c8 I0521 19:32:20.537105 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=5b02f9007ad44055f22ea1bc3594f2c4988e31908d4fd232d63316573af8e10c I0521 19:32:20.537112 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0521 19:32:20.537128 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=891d7b20e277addb799cda4905e7e2ffd5225af7e50dadd9167ff71f7cbfa3bf I0521 19:32:20.537140 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0521 19:32:20.537154 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=77252d44d876daee583c267c5ac7404ea2ca2194fdc730ab0a663eee8b726351 I0521 19:32:20.537168 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0521 19:32:20.537187 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=222526bba0430f041af2e320ccad009c895f5fce6ec52077ce0c346f2a504456 I0521 19:32:20.537197 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0521 19:32:20.537216 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=7d8e38ef1d7f153ea802091634874d5fcf4dfaa007864c220676cb54e1125ef4 I0521 19:32:20.537319 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=535c95881dc0b3dcab0d57fe6da8ab146c73d11140528b16bb69abd476d4b91f I0521 19:32:20.537329 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0521 19:32:20.537335 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0521 19:32:20.537371 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0521 19:32:20.537395 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=4392422b7322eccabead9eb01a9ec086e86836120d48fa4eb1f1b8ab377fecc1 I0521 19:32:20.537422 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=9e652a7ef5aba63183e7985427ca9c450979245ed503c75e47bf64c9e14661f4 I0521 19:32:20.537432 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0521 19:32:20.537447 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=2ba5896ebf2d97636da9e8b37abe00558ddc548306b33535cf5bcc2549e7a7a5 I0521 19:32:20.537457 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0521 19:32:20.537475 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=87a1686c12fea5a634261974b66ec039929612987e03cb88f19510694a80b6e5 I0521 19:32:20.537484 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0521 19:32:20.537497 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=c57beb0b634e69e09da2e59e27f7676d9aab5f586c5ed3a967ca96e4162fd1cf I0521 19:32:20.537514 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=b212c1e183e25fd962b3712c21aa71068050a753cbfb4d5861b24b05a44866b4 I0521 19:32:20.537527 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=3890a443accba938015492edb2125a1d532b486f4a279531303e83e630613c6a I0521 19:32:20.537550 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=ecc8b96159014468ff370570861d8e7a48a02a2438a39df3a40e057a333a75f2 I0521 19:32:20.537560 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0521 19:32:20.537586 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=77579409a57d23ec43bdeebd294a4568fd632a5b7b9c5e036775f6b45e6c72ae I0521 19:32:20.537603 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0521 19:32:20.537610 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0521 19:32:20.537617 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.465733061s to process 39 records W0521 19:32:21.101742 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0521 19:32:22.101773 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0521 19:32:23.099271 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:32:23.099296 1 tasks_processing.go:74] worker 60 stopped. E0521 19:32:23.099307 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:32:23.099319 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0521 19:32:23.099333 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.030273341s to process 1 records I0521 19:32:23.529657 1 controllerstatus.go:89] name=scaController healthy=false reason=Unauthorized message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: OCM API https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates returned HTTP 401: {"code":"ACCT-MGMT-15","href":"/api/accounts_mgmt/v1/errors/15","id":"15","kind":"Error","operation_id":"0f233eee-795d-42a3-9a2b-8e34dddb98f4","reason":"Unable to get payload details from JWT token: Unable to retrieve JWT token from request context"} I0521 19:32:26.685890 1 controllerstatus.go:89] name=clusterTransferController healthy=false reason=Unauthorized message=failed to pull cluster transfer: OCM API https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%274a38241f-a63d-46f2-b323-79020ab8f884%27+and+status+is+%27accepted%27 returned HTTP 401: {"code":"ACCT-MGMT-15","href":"/api/accounts_mgmt/v1/errors/15","id":"15","kind":"Error","operation_id":"a1880246-0c51-4c54-b9d3-9dea53c94a69","reason":"Unable to get payload details from JWT token: Unable to retrieve JWT token from request context"} I0521 19:32:31.115490 1 tasks_processing.go:74] worker 5 stopped. I0521 19:32:31.115530 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0521 19:32:31.115543 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 13.046175697s to process 1 records I0521 19:32:31.879728 1 tasks_processing.go:74] worker 18 stopped. I0521 19:32:31.880034 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=dadc129ba51abfbb756683073832cb0a8ffd78a000cadc82240ae80a16c4db1f I0521 19:32:31.880053 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.808203187s to process 1 records E0521 19:32:31.880315 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.811s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"machine_configs\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error" I0521 19:32:31.881467 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "tsdb_status" failed with an error, function "metrics" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "machine_configs" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error I0521 19:32:31.881745 1 periodic.go:216] Running workloads gatherer I0521 19:32:31.881775 1 tasks_processing.go:45] number of workers: 2 I0521 19:32:31.881787 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:32:31.881792 1 tasks_processing.go:71] worker 1 working on workload_info task. I0521 19:32:31.881797 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:32:31.881811 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0521 19:32:31.890247 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-237.ec2.internal... I0521 19:32:31.890251 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-72.ec2.internal... I0521 19:32:31.890263 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-131.ec2.internal... I0521 19:32:31.916073 1 tasks_processing.go:74] worker 0 stopped. I0521 19:32:31.916091 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 34.248328ms to process 0 records I0521 19:32:36.516099 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 5s I0521 19:32:36.543150 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0521 19:32:36.554394 1 gather_workloads_info.go:387] No image sha256:3a1c7ec9bb0db343348170a70697ea3209ed81e22f1259ff694190e5d04ef250 (12ms) I0521 19:32:36.565339 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (11ms) I0521 19:32:36.575563 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (10ms) I0521 19:32:36.585604 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (10ms) I0521 19:32:36.595533 1 gather_workloads_info.go:387] No image sha256:3380fecebaeee139c1607fbcd9346cebc92806679568e2244a8905ef0f52aad2 (10ms) I0521 19:32:36.606181 1 gather_workloads_info.go:387] No image sha256:1eb6a6ee3c66ac15be6bc21bb6015c595c11c15b9dfd21bd9f5ce7f7b990ebaa (11ms) I0521 19:32:36.629575 1 gather_workloads_info.go:387] No image sha256:53ff8efdd8f563421ebc1859ab021a3bbc1306f82d4ff5d9c21bc60ec5f57898 (23ms) I0521 19:32:36.639377 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (10ms) I0521 19:32:36.649161 1 gather_workloads_info.go:387] No image sha256:15928a5b803fa00781b408dbc4c3bfa7c5287b18584ba31f817def8a4430c725 (10ms) I0521 19:32:36.658777 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (10ms) I0521 19:32:36.668211 1 gather_workloads_info.go:387] No image sha256:572ebf7c7fab1a518582f3ca46c866730423cce6e8a19f614dc15fd61bf0c65a (9ms) I0521 19:32:36.753164 1 gather_workloads_info.go:387] No image sha256:52db05084f2a2e7d7bf4040e4865df88ec4074efbbe59a78d861be2c5ab2aa40 (85ms) I0521 19:32:36.853312 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (100ms) I0521 19:32:36.952873 1 gather_workloads_info.go:387] No image sha256:0b2e6536a0ca3afc5e64a218087a6287c74b77a359075a385f1949ca5bd51b22 (100ms) I0521 19:32:37.052580 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (100ms) I0521 19:32:37.152928 1 gather_workloads_info.go:387] No image sha256:bdeb1c455cffccdb6c90db238e9d371de06340b1d6105d47f66d87293f878ac1 (100ms) I0521 19:32:37.252706 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (100ms) I0521 19:32:37.352209 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (99ms) I0521 19:32:37.453880 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (102ms) I0521 19:32:37.552914 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (99ms) I0521 19:32:37.656156 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (103ms) I0521 19:32:37.754462 1 gather_workloads_info.go:398] Found image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (98ms) I0521 19:32:37.854534 1 gather_workloads_info.go:387] No image sha256:5d6a2b8d2508c7da71192661e7c0dcf1335f48b6336a20189abffbe79eeff962 (100ms) I0521 19:32:37.956524 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (102ms) I0521 19:32:38.053730 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (97ms) I0521 19:32:38.153986 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (100ms) I0521 19:32:38.254099 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (100ms) I0521 19:32:38.359275 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (105ms) I0521 19:32:38.454025 1 gather_workloads_info.go:387] No image sha256:17aefdc2f4752c23dc8f0947bb01fc34635822eeb18f2d8a5eddf8211b84d450 (95ms) I0521 19:32:38.553877 1 gather_workloads_info.go:387] No image sha256:5ff204630794311b0b37fa7b197b933ad85d76a481bf7cdb3bcbada08f0cdcbf (100ms) I0521 19:32:38.654617 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (101ms) I0521 19:32:38.753919 1 gather_workloads_info.go:387] No image sha256:48f5559439bea030a6423652da448c25fff0c62ba293eb521843c1a8eb89ddb0 (99ms) I0521 19:32:38.857572 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (104ms) I0521 19:32:38.953256 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (96ms) I0521 19:32:39.053882 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (101ms) I0521 19:32:39.152752 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (99ms) I0521 19:32:39.255078 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (102ms) I0521 19:32:39.355973 1 gather_workloads_info.go:387] No image sha256:15e1b5f763553e86d0230a751b0783e3b990438fd07cf96eed164bccb72ed564 (101ms) I0521 19:32:39.453536 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (98ms) I0521 19:32:39.553062 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (100ms) I0521 19:32:39.654094 1 gather_workloads_info.go:387] No image sha256:9d6d63dce784df5863e6a0ba4c11317a26c9d756d4e62d47759c97190a71cce5 (101ms) I0521 19:32:39.755515 1 gather_workloads_info.go:387] No image sha256:9ef6dc311b880a6442b7a3a19006ddd955c7da36850bf000f5c9b68fd0862dd4 (101ms) I0521 19:32:39.853655 1 gather_workloads_info.go:387] No image sha256:0acfe677ed58031f8381cb8a4f2d8917ec4cfa95485e2a46529dffae54c75ace (98ms) I0521 19:32:39.955380 1 gather_workloads_info.go:387] No image sha256:6c6d412ccf0c3d6a14645a79df6c4658c0f8725d7954cb83c3d2f0116499345a (102ms) I0521 19:32:40.052908 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (98ms) I0521 19:32:40.154266 1 gather_workloads_info.go:398] Found image sha256:23204877f166455ce8f9de9b91a467eeda576f172d5f96124a68aa986c4775d6 (101ms) I0521 19:32:40.256037 1 gather_workloads_info.go:387] No image sha256:5940cc34b322325618929b1e0eb3864dae73d54c8f5b474e218dfb28123a61c7 (102ms) I0521 19:32:40.256070 1 tasks_processing.go:74] worker 1 stopped. I0521 19:32:40.256419 1 recorder.go:75] Recording config/workload_info with fingerprint=17e256394e1a7b2863f24e4a531d8b425aa05fde7f3528ed72052ae67792df8d I0521 19:32:40.256436 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.374269124s to process 1 records I0521 19:32:40.256454 1 periodic.go:263] Periodic gather workloads completed in 8.374s I0521 19:32:40.256462 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0521 19:32:40.256467 1 periodic.go:216] Running conditional gatherer I0521 19:32:40.263835 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0521 19:32:49.759457 1 conditional_gatherer.go:282] Received HTTP status code 401, trying again in 30s I0521 19:33:19.768260 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0521 19:33:34.823875 1 conditional_gatherer.go:282] Received HTTP status code 504, trying again in 1m0s I0521 19:33:34.823900 1 conditional_gatherer.go:107] received HTTP 504 Gateway Time-out from https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules. Using the default built-in configuration I0521 19:33:34.824261 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0521 19:33:35.902878 1 conditional_gatherer.go:374] alert "ClusterMonitoringOperatorReconciliationErrors" has state "pending" I0521 19:33:35.902896 1 conditional_gatherer.go:374] alert "KubeStatefulSetReplicasMismatch" has state "pending" I0521 19:33:35.902900 1 conditional_gatherer.go:374] alert "KubeContainerWaiting" has state "pending" I0521 19:33:35.902903 1 conditional_gatherer.go:374] alert "KubePdbNotEnoughHealthyPods" has state "pending" I0521 19:33:35.902905 1 conditional_gatherer.go:374] alert "Watchdog" has state "firing" I0521 19:33:35.902909 1 conditional_gatherer.go:374] alert "AlertmanagerMembersInconsistent" has state "pending" I0521 19:33:35.902911 1 conditional_gatherer.go:374] alert "AlertmanagerReceiversNotConfigured" has state "pending" I0521 19:33:35.902988 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0521 19:33:35.912993 1 conditional_gatherer.go:392] cluster version is '4.21.9' I0521 19:33:35.913023 1 tasks_processing.go:45] number of workers: 3 I0521 19:33:35.913034 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:33:35.913039 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0521 19:33:35.913046 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:33:35.913057 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:33:35.913063 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0521 19:33:35.913066 1 tasks_processing.go:74] worker 1 stopped. I0521 19:33:35.913059 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0521 19:33:35.913099 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0521 19:33:35.913112 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 976ns to process 1 records I0521 19:33:35.913167 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=5e6e9deebe7018085c2e386bec070d88cace102ade375a07031a50ddcdc08e82 I0521 19:33:35.913176 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.35µs to process 1 records I0521 19:33:35.913182 1 tasks_processing.go:74] worker 2 stopped. I0521 19:33:35.913324 1 tasks_processing.go:74] worker 0 stopped. I0521 19:33:35.913344 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 245.124µs to process 0 records I0521 19:33:35.913379 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=HttpStatus504 message=received HTTP 504 Gateway Time-out from https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules. Using the default built-in configuration I0521 19:33:35.913397 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0521 19:33:35.942924 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=27229b188eae5f25d901d92f7f003bd299444931fbe47c09ff0587a4fdbe3b2f I0521 19:33:35.943068 1 diskrecorder.go:70] Writing 110 records to /var/lib/insights-operator/insights-2026-05-21-193335.tar.gz I0521 19:33:35.951417 1 diskrecorder.go:51] Wrote 110 records to disk in 8ms I0521 19:33:35.951445 1 periodic.go:285] Gathering cluster info every 2h0m0s I0521 19:33:35.951460 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0521 19:33:48.069494 1 diskrecorder.go:223] Found files to send: insights-2026-05-21-193335.tar.gz I0521 19:33:48.069526 1 insightsuploader.go:150] Checking archives to upload periodically every 15m30.042741201s I0521 19:33:48.069535 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0521 19:33:48.082544 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0521 19:34:03.155250 1 requests.go:69] gateway server https://console.redhat.com/api/ingress/v1/upload returned 401, x-rh-insights-request-id=92e9dcfa58904f4b953b7760d1bd505a I0521 19:34:03.155334 1 insightsuploader.go:169] Unable to upload report after 15.08s: your Red Hat account is not enabled for remote support or your token has expired: {"errors":[{"meta":{"response_by":"gateway"},"status":401,"detail":"UHC services authentication failed"}]} I0521 19:34:03.155371 1 controllerstatus.go:80] name=insightsuploader healthy=false reason=NotAuthorized message=Reporting was not allowed: your Red Hat account is not enabled for remote support or your token has expired: {"errors":[{"meta":{"response_by":"gateway"},"status":401,"detail":"UHC services authentication failed"}]} I0521 19:34:18.085243 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:34:18.085278 1 controller.go:554] The operator is healthy I0521 19:36:18.098246 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:36:18.098292 1 controller.go:554] The operator is healthy I0521 19:37:18.060299 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 19:37:18.069448 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:37:18.069466 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 19:37:18.074485 1 secretconfigobserver.go:119] support secret does not exist I0521 19:38:18.084408 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:38:18.084483 1 controller.go:554] The operator is healthy I0521 19:40:18.084452 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:40:18.084493 1 controller.go:554] The operator is healthy I0521 19:42:18.050087 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:42:18.058964 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:42:18.075050 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 19:42:18.080510 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:42:18.080527 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 19:42:18.086578 1 secretconfigobserver.go:119] support secret does not exist I0521 19:42:18.088082 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:42:18.088114 1 controller.go:554] The operator is healthy I0521 19:44:18.084172 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:44:18.084216 1 controller.go:554] The operator is healthy I0521 19:46:18.084948 1 controller.go:271] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:46:18.084985 1 controller.go:554] The operator is healthy I0521 19:47:18.087080 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 19:47:18.093539 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:47:18.093557 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 19:47:18.098396 1 secretconfigobserver.go:119] support secret does not exist