I0504 20:37:46.507058 1 cmd.go:253] Using service-serving-cert provided certificates I0504 20:37:46.507485 1 observer_polling.go:159] Starting file observer I0504 20:37:47.335810 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0504 20:37:47.336021 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0504 20:37:47.336618 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0504 20:37:47.337099 1 secure_serving.go:57] Forcing use of http/1.1 only W0504 20:37:47.337119 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0504 20:37:47.337124 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0504 20:37:47.337129 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0504 20:37:47.337132 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0504 20:37:47.337135 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0504 20:37:47.337137 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0504 20:37:47.340610 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0504 20:37:47.340692 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"6d32edba-ddd0-46f1-8750-496328cac365", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0504 20:37:47.341850 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0504 20:37:47.341854 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0504 20:37:47.341869 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0504 20:37:47.341884 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0504 20:37:47.341888 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0504 20:37:47.341883 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0504 20:37:47.342174 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0504 20:37:47.342561 1 secure_serving.go:213] Serving securely on [::]:8443 I0504 20:37:47.342591 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0504 20:37:47.349806 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0504 20:37:47.349831 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0504 20:37:47.349862 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0504 20:37:47.356023 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 20:37:47.356040 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0504 20:37:47.359530 1 secretconfigobserver.go:119] support secret does not exist I0504 20:37:47.363424 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 20:37:47.366832 1 secretconfigobserver.go:119] support secret does not exist I0504 20:37:47.370087 1 recorder.go:176] Pruning old reports every 5h42m27s, max age is 288h0m0s I0504 20:37:47.374251 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0504 20:37:47.374264 1 periodic.go:216] Running clusterconfig gatherer I0504 20:37:47.374269 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0504 20:37:47.374289 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0504 20:37:47.374294 1 insightsreport.go:296] Starting report retriever I0504 20:37:47.374298 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0504 20:37:47.374333 1 tasks_processing.go:45] number of workers: 64 I0504 20:37:47.374353 1 tasks_processing.go:69] worker 3 listening for tasks. I0504 20:37:47.374364 1 tasks_processing.go:69] worker 2 listening for tasks. I0504 20:37:47.374370 1 tasks_processing.go:69] worker 29 listening for tasks. I0504 20:37:47.374370 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 20:37:47.374378 1 tasks_processing.go:69] worker 30 listening for tasks. I0504 20:37:47.374380 1 tasks_processing.go:69] worker 4 listening for tasks. I0504 20:37:47.374384 1 tasks_processing.go:71] worker 30 working on sap_datahubs task. I0504 20:37:47.374389 1 tasks_processing.go:69] worker 5 listening for tasks. I0504 20:37:47.374389 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 20:37:47.374372 1 tasks_processing.go:71] worker 2 working on schedulers task. I0504 20:37:47.374392 1 tasks_processing.go:69] worker 47 listening for tasks. I0504 20:37:47.374399 1 tasks_processing.go:69] worker 7 listening for tasks. I0504 20:37:47.374403 1 tasks_processing.go:69] worker 8 listening for tasks. I0504 20:37:47.374408 1 tasks_processing.go:69] worker 9 listening for tasks. I0504 20:37:47.374414 1 tasks_processing.go:69] worker 39 listening for tasks. I0504 20:37:47.374413 1 tasks_processing.go:69] worker 38 listening for tasks. I0504 20:37:47.374418 1 tasks_processing.go:69] worker 11 listening for tasks. I0504 20:37:47.374422 1 tasks_processing.go:69] worker 40 listening for tasks. I0504 20:37:47.374414 1 tasks_processing.go:69] worker 10 listening for tasks. I0504 20:37:47.374427 1 tasks_processing.go:69] worker 34 listening for tasks. I0504 20:37:47.374429 1 tasks_processing.go:69] worker 12 listening for tasks. I0504 20:37:47.374436 1 tasks_processing.go:69] worker 41 listening for tasks. I0504 20:37:47.374439 1 tasks_processing.go:69] worker 14 listening for tasks. I0504 20:37:47.374433 1 tasks_processing.go:69] worker 13 listening for tasks. I0504 20:37:47.374440 1 tasks_processing.go:69] worker 57 listening for tasks. I0504 20:37:47.374448 1 tasks_processing.go:69] worker 15 listening for tasks. I0504 20:37:47.374446 1 tasks_processing.go:69] worker 37 listening for tasks. I0504 20:37:47.374430 1 tasks_processing.go:69] worker 35 listening for tasks. I0504 20:37:47.374453 1 tasks_processing.go:69] worker 46 listening for tasks. I0504 20:37:47.374436 1 tasks_processing.go:69] worker 36 listening for tasks. I0504 20:37:47.374460 1 tasks_processing.go:69] worker 49 listening for tasks. I0504 20:37:47.374462 1 tasks_processing.go:69] worker 33 listening for tasks. I0504 20:37:47.374466 1 tasks_processing.go:69] worker 24 listening for tasks. I0504 20:37:47.374468 1 tasks_processing.go:69] worker 62 listening for tasks. I0504 20:37:47.374463 1 tasks_processing.go:69] worker 44 listening for tasks. I0504 20:37:47.374473 1 tasks_processing.go:69] worker 25 listening for tasks. I0504 20:37:47.374476 1 tasks_processing.go:69] worker 54 listening for tasks. I0504 20:37:47.374469 1 tasks_processing.go:69] worker 50 listening for tasks. I0504 20:37:47.374481 1 tasks_processing.go:69] worker 60 listening for tasks. I0504 20:37:47.374482 1 tasks_processing.go:69] worker 43 listening for tasks. I0504 20:37:47.374487 1 tasks_processing.go:69] worker 52 listening for tasks. I0504 20:37:47.374489 1 tasks_processing.go:69] worker 61 listening for tasks. I0504 20:37:47.374446 1 tasks_processing.go:69] worker 31 listening for tasks. I0504 20:37:47.374494 1 tasks_processing.go:71] worker 3 working on pod_network_connectivity_checks task. I0504 20:37:47.374496 1 tasks_processing.go:69] worker 18 listening for tasks. I0504 20:37:47.374498 1 tasks_processing.go:69] worker 20 listening for tasks. I0504 20:37:47.374501 1 tasks_processing.go:69] worker 21 listening for tasks. I0504 20:37:47.374502 1 tasks_processing.go:69] worker 58 listening for tasks. I0504 20:37:47.374505 1 tasks_processing.go:71] worker 4 working on nodes task. I0504 20:37:47.374510 1 tasks_processing.go:71] worker 14 working on image_pruners task. I0504 20:37:47.374503 1 tasks_processing.go:69] worker 51 listening for tasks. I0504 20:37:47.374516 1 tasks_processing.go:71] worker 13 working on storage_cluster task. I0504 20:37:47.374516 1 tasks_processing.go:71] worker 10 working on nodenetworkstates task. I0504 20:37:47.374523 1 tasks_processing.go:71] worker 37 working on version task. I0504 20:37:47.374526 1 tasks_processing.go:71] worker 38 working on proxies task. I0504 20:37:47.374528 1 tasks_processing.go:71] worker 49 working on feature_gates task. I0504 20:37:47.374516 1 tasks_processing.go:71] worker 5 working on networks task. I0504 20:37:47.374538 1 tasks_processing.go:71] worker 47 working on openshift_machine_api_events task. I0504 20:37:47.374534 1 tasks_processing.go:71] worker 9 working on node_features task. I0504 20:37:47.374546 1 tasks_processing.go:71] worker 60 working on clusterroles task. I0504 20:37:47.374580 1 tasks_processing.go:71] worker 31 working on cluster_apiserver task. I0504 20:37:47.374579 1 tasks_processing.go:71] worker 40 working on container_runtime_configs task. I0504 20:37:47.374586 1 tasks_processing.go:71] worker 43 working on certificate_signing_requests task. I0504 20:37:47.374374 1 tasks_processing.go:71] worker 29 working on nodenetworkconfigurationpolicies task. I0504 20:37:47.374592 1 tasks_processing.go:71] worker 18 working on active_alerts task. I0504 20:37:47.374598 1 tasks_processing.go:71] worker 44 working on aggregated_monitoring_cr_names task. I0504 20:37:47.374524 1 tasks_processing.go:71] worker 36 working on image_registries task. I0504 20:37:47.374447 1 tasks_processing.go:69] worker 45 listening for tasks. I0504 20:37:47.374672 1 tasks_processing.go:71] worker 45 working on storage_classes task. I0504 20:37:47.374524 1 tasks_processing.go:71] worker 51 working on validating_webhook_configurations task. I0504 20:37:47.374509 1 tasks_processing.go:71] worker 11 working on dvo_metrics task. I0504 20:37:47.374395 1 tasks_processing.go:69] worker 6 listening for tasks. I0504 20:37:47.374752 1 tasks_processing.go:71] worker 61 working on container_images task. I0504 20:37:47.374757 1 tasks_processing.go:71] worker 24 working on lokistack task. I0504 20:37:47.374761 1 tasks_processing.go:71] worker 33 working on openshift_logging task. I0504 20:37:47.374767 1 tasks_processing.go:71] worker 25 working on crds task. I0504 20:37:47.374785 1 tasks_processing.go:71] worker 50 working on mutating_webhook_configurations task. I0504 20:37:47.374805 1 tasks_processing.go:71] worker 6 working on monitoring_persistent_volumes task. I0504 20:37:47.374452 1 tasks_processing.go:69] worker 32 listening for tasks. I0504 20:37:47.374458 1 tasks_processing.go:69] worker 23 listening for tasks. I0504 20:37:47.374818 1 tasks_processing.go:71] worker 32 working on jaegers task. I0504 20:37:47.374465 1 tasks_processing.go:69] worker 26 listening for tasks. I0504 20:37:47.374473 1 tasks_processing.go:69] worker 27 listening for tasks. I0504 20:37:47.374474 1 tasks_processing.go:69] worker 42 listening for tasks. I0504 20:37:47.374475 1 tasks_processing.go:69] worker 22 listening for tasks. I0504 20:37:47.374477 1 tasks_processing.go:69] worker 28 listening for tasks. I0504 20:37:47.374486 1 tasks_processing.go:69] worker 55 listening for tasks. I0504 20:37:47.374489 1 tasks_processing.go:69] worker 19 listening for tasks. I0504 20:37:47.374491 1 tasks_processing.go:69] worker 17 listening for tasks. I0504 20:37:47.374496 1 tasks_processing.go:69] worker 59 listening for tasks. I0504 20:37:47.374486 1 tasks_processing.go:69] worker 53 listening for tasks. I0504 20:37:47.374492 1 tasks_processing.go:69] worker 56 listening for tasks. I0504 20:37:47.374498 1 tasks_processing.go:69] worker 63 listening for tasks. I0504 20:37:47.374511 1 tasks_processing.go:71] worker 34 working on silenced_alerts task. I0504 20:37:47.374532 1 tasks_processing.go:71] worker 7 working on machine_configs task. I0504 20:37:47.374454 1 tasks_processing.go:69] worker 16 listening for tasks. I0504 20:37:47.374516 1 tasks_processing.go:71] worker 58 working on oauths task. I0504 20:37:47.374996 1 tasks_processing.go:71] worker 16 working on install_plans task. I0504 20:37:47.375040 1 tasks_processing.go:71] worker 53 working on openstack_version task. I0504 20:37:47.374379 1 tasks_processing.go:71] worker 1 working on ingress task. I0504 20:37:47.374514 1 tasks_processing.go:71] worker 35 working on metrics task. I0504 20:37:47.374510 1 tasks_processing.go:71] worker 57 working on ceph_cluster task. I0504 20:37:47.374520 1 tasks_processing.go:71] worker 46 working on support_secret task. I0504 20:37:47.374521 1 tasks_processing.go:71] worker 39 working on machine_healthchecks task. I0504 20:37:47.374563 1 tasks_processing.go:71] worker 12 working on image task. I0504 20:37:47.374563 1 tasks_processing.go:71] worker 8 working on olm_operators task. I0504 20:37:47.374569 1 tasks_processing.go:71] worker 41 working on openstack_dataplanenodesets task. I0504 20:37:47.374581 1 tasks_processing.go:71] worker 52 working on sap_config task. I0504 20:37:47.374588 1 tasks_processing.go:71] worker 20 working on openstack_controlplanes task. I0504 20:37:47.374715 1 tasks_processing.go:71] worker 21 working on machine_autoscalers task. I0504 20:37:47.374455 1 tasks_processing.go:69] worker 48 listening for tasks. I0504 20:37:47.378117 1 tasks_processing.go:71] worker 48 working on config_maps task. I0504 20:37:47.374762 1 tasks_processing.go:71] worker 54 working on qemu_kubevirt_launcher_logs task. I0504 20:37:47.374769 1 tasks_processing.go:71] worker 62 working on node_logs task. I0504 20:37:47.375011 1 tasks_processing.go:71] worker 55 working on infrastructures task. I0504 20:37:47.374518 1 tasks_processing.go:71] worker 15 working on operators_pods_and_events task. I0504 20:37:47.375017 1 tasks_processing.go:71] worker 23 working on openstack_dataplanedeployments task. I0504 20:37:47.375021 1 tasks_processing.go:71] worker 22 working on sap_pods task. I0504 20:37:47.375022 1 tasks_processing.go:71] worker 42 working on ingress_certificates task. I0504 20:37:47.375018 1 tasks_processing.go:71] worker 59 working on operators task. I0504 20:37:47.375024 1 tasks_processing.go:71] worker 28 working on machine_sets task. I0504 20:37:47.375024 1 tasks_processing.go:71] worker 26 working on tsdb_status task. I0504 20:37:47.375031 1 tasks_processing.go:71] worker 19 working on machines task. I0504 20:37:47.375027 1 tasks_processing.go:71] worker 27 working on service_accounts task. I0504 20:37:47.375035 1 tasks_processing.go:71] worker 56 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0504 20:37:47.375034 1 tasks_processing.go:71] worker 17 working on cost_management_metrics_configs task. I0504 20:37:47.375037 1 tasks_processing.go:71] worker 63 working on overlapping_namespace_uids task. I0504 20:37:47.374515 1 tasks_processing.go:71] worker 0 working on authentication task. E0504 20:37:47.380224 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0504 20:37:47.380245 1 tasks_processing.go:71] worker 9 working on pdbs task. I0504 20:37:47.380445 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 5.68991ms to process 0 records E0504 20:37:47.380475 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0504 20:37:47.380484 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 5.953516ms to process 0 records I0504 20:37:47.380499 1 tasks_processing.go:71] worker 3 working on machine_config_pools task. I0504 20:37:47.380659 1 tasks_processing.go:74] worker 29 stopped. I0504 20:37:47.380690 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 6.055289ms to process 0 records I0504 20:37:47.380753 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 6.129421ms to process 0 records I0504 20:37:47.380826 1 tasks_processing.go:74] worker 40 stopped. I0504 20:37:47.380842 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 6.231056ms to process 0 records I0504 20:37:47.380857 1 tasks_processing.go:74] worker 30 stopped. I0504 20:37:47.380876 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 6.465273ms to process 0 records I0504 20:37:47.380918 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 6.360591ms to process 0 records I0504 20:37:47.380927 1 tasks_processing.go:74] worker 13 stopped. I0504 20:37:47.381076 1 tasks_processing.go:74] worker 38 stopped. I0504 20:37:47.381231 1 recorder.go:75] Recording config/proxy with fingerprint=14879993d9fe34b21c90fe7a97b99b9456811011f2d0070b158e3eb3e90dd075 I0504 20:37:47.381251 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 6.53908ms to process 1 records I0504 20:37:47.381266 1 tasks_processing.go:74] worker 2 stopped. I0504 20:37:47.381275 1 tasks_processing.go:74] worker 10 stopped. I0504 20:37:47.381426 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=17885087fb5bfb182e08d4a1a5551f53d2fb6ba76e638cbf0767ad6692ea545f I0504 20:37:47.381498 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 6.863923ms to process 1 records I0504 20:37:47.382210 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0504 20:37:47.382232 1 controller.go:368] The initial operator extension status is healthy I0504 20:37:47.382243 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0504 20:37:47.382250 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0504 20:37:47.382254 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0504 20:37:47.382276 1 controller.go:531] The operator is still being initialized I0504 20:37:47.382289 1 controller.go:554] The operator is healthy I0504 20:37:47.382383 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0504 20:37:47.382434 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0504 20:37:47.382527 1 operator.go:328] started I0504 20:37:47.382555 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer E0504 20:37:47.382908 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.162.146:9091: connect: connection refused I0504 20:37:47.382934 1 tasks_processing.go:74] worker 26 stopped. E0504 20:37:47.382941 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.162.146:9091: connect: connection refused I0504 20:37:47.382950 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 3.82285ms to process 0 records E0504 20:37:47.382979 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.162.146:9091: connect: connection refused I0504 20:37:47.383010 1 tasks_processing.go:74] worker 35 stopped. E0504 20:37:47.383021 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.162.146:9091: connect: connection refused I0504 20:37:47.383042 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 7.653202ms to process 0 records I0504 20:37:47.384517 1 tasks_processing.go:74] worker 31 stopped. I0504 20:37:47.384651 1 recorder.go:75] Recording config/apiserver with fingerprint=e21e38859fca6fdd22dca89fafe35d6e25ebc44559fa770d700be34b295adc08 I0504 20:37:47.384663 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 9.9221ms to process 1 records I0504 20:37:47.385899 1 tasks_processing.go:74] worker 32 stopped. I0504 20:37:47.385908 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 11.070077ms to process 0 records I0504 20:37:47.391746 1 tasks_processing.go:74] worker 57 stopped. I0504 20:37:47.391759 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 16.343098ms to process 0 records I0504 20:37:47.395491 1 tasks_processing.go:74] worker 39 stopped. E0504 20:37:47.395501 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0504 20:37:47.395510 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 19.99296ms to process 0 records I0504 20:37:47.395728 1 tasks_processing.go:74] worker 49 stopped. I0504 20:37:47.395911 1 recorder.go:75] Recording config/featuregate with fingerprint=7b5fcad45f0135aed0a3e6db7003f36659e03c042ac3f9273296325d65b9ff07 I0504 20:37:47.395930 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 21.182103ms to process 1 records I0504 20:37:47.406529 1 tasks_processing.go:74] worker 53 stopped. I0504 20:37:47.406544 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 31.395602ms to process 0 records I0504 20:37:47.406563 1 tasks_processing.go:74] worker 24 stopped. I0504 20:37:47.406568 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 31.795575ms to process 0 records I0504 20:37:47.406577 1 tasks_processing.go:74] worker 41 stopped. I0504 20:37:47.406589 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 30.93858ms to process 0 records I0504 20:37:47.406622 1 tasks_processing.go:74] worker 33 stopped. I0504 20:37:47.406636 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 31.846669ms to process 0 records I0504 20:37:47.406667 1 tasks_processing.go:74] worker 20 stopped. I0504 20:37:47.406684 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 28.750637ms to process 0 records I0504 20:37:47.406696 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 30.948204ms to process 0 records I0504 20:37:47.406704 1 tasks_processing.go:74] worker 52 stopped. I0504 20:37:47.407007 1 tasks_processing.go:74] worker 5 stopped. I0504 20:37:47.407148 1 recorder.go:75] Recording config/network with fingerprint=e7a5c27dfd109d871e34337a77a7e5efd323daf641b160b3dbde682feb1d608f I0504 20:37:47.407163 1 gather.go:177] gatherer "clusterconfig" function "networks" took 32.458774ms to process 1 records I0504 20:37:47.407250 1 tasks_processing.go:74] worker 14 stopped. I0504 20:37:47.407422 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=60405bf81a62337d1fb85ae5cdc11f1bccaca376dfb52a195393ada9ac98a54a I0504 20:37:47.407435 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 32.626889ms to process 1 records I0504 20:37:47.409012 1 tasks_processing.go:74] worker 58 stopped. I0504 20:37:47.409285 1 recorder.go:75] Recording config/oauth with fingerprint=79ee8e45dc7111f340421a9f6b004e3b15a5477f4941980a99078567c6af86cd I0504 20:37:47.409325 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 34.096666ms to process 1 records I0504 20:37:47.409559 1 tasks_processing.go:74] worker 12 stopped. I0504 20:37:47.409682 1 recorder.go:75] Recording config/image with fingerprint=809e4baf50db8e17899533a304ecff9e710ef098bf9cc9f7c2780d5e65448ce4 I0504 20:37:47.409700 1 gather.go:177] gatherer "clusterconfig" function "image" took 34.029479ms to process 1 records I0504 20:37:47.409709 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 34.021884ms to process 0 records I0504 20:37:47.409716 1 tasks_processing.go:74] worker 8 stopped. I0504 20:37:47.409733 1 tasks_processing.go:74] worker 6 stopped. I0504 20:37:47.409745 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 34.9189ms to process 0 records I0504 20:37:47.410677 1 tasks_processing.go:74] worker 23 stopped. I0504 20:37:47.410696 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 32.095721ms to process 0 records I0504 20:37:47.410705 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 31.64283ms to process 0 records I0504 20:37:47.410715 1 tasks_processing.go:74] worker 28 stopped. I0504 20:37:47.410723 1 tasks_processing.go:74] worker 46 stopped. E0504 20:37:47.410737 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0504 20:37:47.410746 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 35.24173ms to process 0 records I0504 20:37:47.411044 1 gather_logs.go:145] no pods in namespace were found I0504 20:37:47.411059 1 tasks_processing.go:74] worker 54 stopped. I0504 20:37:47.411065 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 32.833119ms to process 0 records I0504 20:37:47.411112 1 tasks_processing.go:74] worker 50 stopped. I0504 20:37:47.411371 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=840c6306c5455f0a241b00c15496035ae9c4f714381bad5bdde8d9aa04cca44d I0504 20:37:47.411438 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=14d800d961a9a7d296de8162e3f90042369cbdf85f142e4b426b5b9a344d1b91 I0504 20:37:47.411494 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=b726910193a06cedcf5ccfcd55cdc558a914dd7d795cc9d6938cdecaed34d1e1 I0504 20:37:47.411518 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 36.317141ms to process 3 records I0504 20:37:47.411616 1 tasks_processing.go:74] worker 1 stopped. I0504 20:37:47.411680 1 recorder.go:75] Recording config/ingress with fingerprint=d36bcabe20a27e6f05fc6255bf9fc4eac7f7023bfd6a89497c1afefcabc8a6eb I0504 20:37:47.411697 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 36.005974ms to process 1 records I0504 20:37:47.411704 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 36.737629ms to process 0 records E0504 20:37:47.411709 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0504 20:37:47.411715 1 gather.go:177] gatherer "clusterconfig" function "machines" took 32.190249ms to process 0 records I0504 20:37:47.411721 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 32.666382ms to process 0 records I0504 20:37:47.411725 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 32.185899ms to process 0 records I0504 20:37:47.411731 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 30.98182ms to process 0 records I0504 20:37:47.411730 1 tasks_processing.go:74] worker 19 stopped. I0504 20:37:47.411736 1 tasks_processing.go:74] worker 47 stopped. I0504 20:37:47.411742 1 tasks_processing.go:74] worker 3 stopped. I0504 20:37:47.411744 1 tasks_processing.go:74] worker 17 stopped. I0504 20:37:47.411747 1 tasks_processing.go:74] worker 22 stopped. E0504 20:37:47.411737 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0504 20:37:47.411778 1 tasks_processing.go:74] worker 7 stopped. I0504 20:37:47.411779 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0504 20:37:47.411798 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 36.596916ms to process 1 records I0504 20:37:47.411804 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 33.552433ms to process 0 records I0504 20:37:47.411830 1 tasks_processing.go:74] worker 21 stopped. I0504 20:37:47.411891 1 tasks_processing.go:74] worker 45 stopped. I0504 20:37:47.411928 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=4312aa685353dadd8a6d6defce430228313790d8d47e5aca8731efaa1a7dd7c1 I0504 20:37:47.411945 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=f21e8a3e8b753f8ca15f8ef0aa9df4e5a2a48d8a8341b50973992daea4f51b61 I0504 20:37:47.411973 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 36.949042ms to process 2 records I0504 20:37:47.412085 1 tasks_processing.go:74] worker 55 stopped. I0504 20:37:47.414067 1 recorder.go:75] Recording config/infrastructure with fingerprint=c833e9d9c0f48fdeccc6fcc7e82d388dcff12eb73444815b49e861bc11ba9ef7 I0504 20:37:47.414090 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 33.435484ms to process 1 records I0504 20:37:47.414514 1 tasks_processing.go:74] worker 0 stopped. I0504 20:37:47.414579 1 recorder.go:75] Recording config/authentication with fingerprint=9c0f4cc720d8955bc889ecad3df5c2fb196536593826f37b779ec46ac9d681bc I0504 20:37:47.414645 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 32.649107ms to process 1 records I0504 20:37:47.414757 1 tasks_processing.go:74] worker 36 stopped. I0504 20:37:47.415747 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=dc9a4ffe6b542a8f3a2fd9cfe3e9d3674f58da00d079e3a61fc1648e593fb213 I0504 20:37:47.415764 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 37.537187ms to process 1 records I0504 20:37:47.415856 1 tasks_processing.go:74] worker 51 stopped. I0504 20:37:47.415865 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=aebd7b4faa8f9bd390d8416361869234a2a61258010c85b1464a58dced07284b I0504 20:37:47.415942 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=0676d4ef191e75007a92e98e57fd0309dad8eb6a46c4ca818f23484c3a946843 I0504 20:37:47.416067 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=55c5a881fcb2ff7280d97e63ee2ad4c1b053b34a69fc1b46475ad82bac2d7b54 I0504 20:37:47.416111 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=6693e18cb67f1ea675d45a053865f99ba4c7e037b9cb5e462cfcaf6f2353e18a I0504 20:37:47.416147 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=54376c36d4941c1ef1a5ac0164fd54856769aa2d10679346568b0274b35aee51 I0504 20:37:47.416197 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=22e03ba825de6f72e583b4a71996430f7dbdf3ed80fcc82ace66eb96e7cadf78 I0504 20:37:47.416203 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0504 20:37:47.416210 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0504 20:37:47.416215 1 controller.go:254] Source scaController *sca.Controller is not ready I0504 20:37:47.416220 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0504 20:37:47.416224 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0504 20:37:47.416247 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=a4e4d006a87065aa41cf7c2af30aa6ed5a98916c97e2e7406d197ed619f56219 I0504 20:37:47.416250 1 controller.go:531] The operator is still being initialized I0504 20:37:47.416261 1 controller.go:554] The operator is healthy I0504 20:37:47.416297 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=393c5973a3b89b2fd4ee76aa9126a990f569f6f5aab5b16b8f178f409dffec8e I0504 20:37:47.416347 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=671eae186aaeadb33728be1884f6f0f4cfc65de9cff74b6790c50d460e6c88cf I0504 20:37:47.416386 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=a451008f514fdf1d178a6e0417bbed7e6fc7eb253e4320b55d5d9fd797f27561 I0504 20:37:47.416442 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=0d6884e05e6b2f88f7e1e7a6b6daab53f185270cd755b473e2886d94cf0359ac I0504 20:37:47.416480 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=29ac5afbe3d77ae1cef6ef273162b427dd435eeb71a4f1838a44a24604920878 I0504 20:37:47.416510 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=c83ecd34fe47f36a1284f6bcd1fceb7bb84a7a451b761419ec34ed41e69bff8f I0504 20:37:47.416518 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 39.177539ms to process 13 records I0504 20:37:47.416588 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=183fa6b9e35c31d3586c1165d93a9606b03b11f1b8b8ee46ff3bcba2e57af08e I0504 20:37:47.416598 1 tasks_processing.go:74] worker 9 stopped. I0504 20:37:47.416604 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=241743bd0f1be5a74dc47c192a06a6575468f83a537fb94ad3cfd12e38d9604a I0504 20:37:47.416615 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=bd401daa9bf2e43f34aeef6f0c9d0ee14b3afd8e3febabb4027890c07715c606 I0504 20:37:47.416627 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=1da19ae9250c78260097a214de4f544e2f1261e5a5ddbd73529db5b758a22b88 I0504 20:37:47.416645 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=8fc63458300263fdf6977fc07ef0c6274fa0544416132bbcd43b2f5891b5fe4d I0504 20:37:47.416657 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=08cde850046624ccd439b05c14fa6965bf56c84900e016f900d122bef18e9a43 I0504 20:37:47.416670 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=bfadf43ecacb03841b73a0c24f9b40fc41c459df72cf5ddf5d0376333e51b3ad I0504 20:37:47.416699 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=33c21d7056cf3ead054ca6ec203cd056c2a9702e75daae8ee2477485c25cca0c I0504 20:37:47.416714 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=2a42144acc3b0039e34959cdc2cceb0bf2cd3f3e6649f706afd3b2e6b9089e7b I0504 20:37:47.416728 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=5593ae26833112759cb20d723ee082c56beb65d1be1dd14f5e854e1e5b0fd04d I0504 20:37:47.416765 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=3092e2eb895e3be04822a23e3ef9d5b927dfe74389d73c6abbf265a745162632 I0504 20:37:47.416772 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 35.288424ms to process 11 records I0504 20:37:47.416857 1 tasks_processing.go:74] worker 4 stopped. I0504 20:37:47.417034 1 recorder.go:75] Recording config/node/ip-10-0-0-243.ec2.internal with fingerprint=36ea5996b0ef8b97f56e09c3c4d57814220268cf3bef4ec9b8f6ba61c27c8d49 I0504 20:37:47.417091 1 recorder.go:75] Recording config/node/ip-10-0-1-160.ec2.internal with fingerprint=950960ad28638fa1e58b7343139dcd5ab4d2490b53ff939dcb1ed8789cf00a9b I0504 20:37:47.417141 1 recorder.go:75] Recording config/node/ip-10-0-2-50.ec2.internal with fingerprint=1306a55603827264d9c31a64c0549553a09087190b01b99efb5f76fb8ff8ce7a I0504 20:37:47.417150 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 41.125273ms to process 3 records I0504 20:37:47.417463 1 tasks_processing.go:74] worker 43 stopped. I0504 20:37:47.417477 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 42.858787ms to process 0 records I0504 20:37:47.419798 1 tasks_processing.go:74] worker 15 stopped. I0504 20:37:47.419823 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 41.270646ms to process 0 records W0504 20:37:47.422789 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 20:37:47.424768 1 tasks_processing.go:74] worker 25 stopped. I0504 20:37:47.425235 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=ca5402cfa893e12bdfe556d22166ef3c187cfe935ab518255194d7386d156b90 I0504 20:37:47.425455 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=3f1d990eb77ca380ea090da001ebd731bb56920e54a581f417b254b4e83e63db I0504 20:37:47.425466 1 gather.go:177] gatherer "clusterconfig" function "crds" took 49.98567ms to process 2 records I0504 20:37:47.429249 1 tasks_processing.go:74] worker 60 stopped. I0504 20:37:47.429610 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=820aa48c5508e691f537ba263748854d18d410f0f5a7d0b33ed014ff1a93fbec I0504 20:37:47.429758 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=84e60fc267457d9c436607fe74329ef515f254e23a9f97aef43ec06404e84e62 I0504 20:37:47.429771 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 54.661655ms to process 2 records I0504 20:37:47.430944 1 tasks_processing.go:74] worker 44 stopped. I0504 20:37:47.430961 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 56.330543ms to process 0 records I0504 20:37:47.439406 1 tasks_processing.go:74] worker 61 stopped. I0504 20:37:47.440687 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-66dbbb6468-rh79w with fingerprint=728d8a8575f7f6f629a01a5f3f46e925328a37a8841c8d309b6afab6b8d4affa I0504 20:37:47.440763 1 recorder.go:75] Recording config/running_containers with fingerprint=8111258afea390ca0d13c05a9a8fc19d7f9e1bb949b813d613d38edf0c79a954 I0504 20:37:47.440773 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 64.634606ms to process 2 records E0504 20:37:47.440785 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0504 20:37:47.440793 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q3b2ja7v3odr0esm6oj4gf2cqo48sri-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q3b2ja7v3odr0esm6oj4gf2cqo48sri-primary-cert-bundle-secret" not found I0504 20:37:47.440798 1 tasks_processing.go:74] worker 42 stopped. I0504 20:37:47.440825 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=776a027573e506889aaf6fe0f60f3b6bb3a562ad7ce1ed5f6aeba1c693a60963 I0504 20:37:47.440835 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 60.892523ms to process 1 records I0504 20:37:47.442225 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0504 20:37:47.442233 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0504 20:37:47.442262 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0504 20:37:47.443549 1 tasks_processing.go:74] worker 48 stopped. E0504 20:37:47.443565 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0504 20:37:47.443575 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0504 20:37:47.443579 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0504 20:37:47.443587 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0504 20:37:47.443625 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0504 20:37:47.443633 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0504 20:37:47.443644 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0504 20:37:47.443655 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0504 20:37:47.443706 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0504 20:37:47.443713 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0504 20:37:47.443719 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 65.393785ms to process 7 records I0504 20:37:47.450119 1 base_controller.go:82] Caches are synced for ConfigController I0504 20:37:47.450135 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0504 20:37:47.453343 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:37:47.482676 1 base_controller.go:82] Caches are synced for LoggingSyncer I0504 20:37:47.482690 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0504 20:37:47.486679 1 tasks_processing.go:74] worker 62 stopped. I0504 20:37:47.486692 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 108.343341ms to process 0 records I0504 20:37:47.494344 1 tasks_processing.go:74] worker 37 stopped. I0504 20:37:47.494594 1 recorder.go:75] Recording config/version with fingerprint=503c2732cb3faef0b0ac5730a40d2e3fbdc0180912fe65c8eeb438b436bcc213 I0504 20:37:47.494607 1 recorder.go:75] Recording config/id with fingerprint=95f4e28a5dc227df5edc3966e4fd5af6168a865c5641d31c4f2a8d45dbf538d0 I0504 20:37:47.494614 1 gather.go:177] gatherer "clusterconfig" function "version" took 119.810483ms to process 2 records I0504 20:37:47.494842 1 tasks_processing.go:74] worker 63 stopped. I0504 20:37:47.494862 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0504 20:37:47.494870 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 115.457248ms to process 1 records I0504 20:37:47.495588 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0504 20:37:47.516639 1 tasks_processing.go:74] worker 56 stopped. I0504 20:37:47.516654 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 137.393877ms to process 0 records I0504 20:37:47.550778 1 cluster_transfer.go:102] no available accepted cluster transfer I0504 20:37:47.550793 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0504 20:37:47.669658 1 sca.go:266] etc-pki-entitlement secret successfully created I0504 20:37:47.669684 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated W0504 20:37:48.412378 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 20:37:48.429089 1 tasks_processing.go:74] worker 34 stopped. I0504 20:37:48.429114 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0504 20:37:48.429126 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.054220736s to process 1 records I0504 20:37:48.429612 1 tasks_processing.go:74] worker 18 stopped. I0504 20:37:48.429651 1 recorder.go:75] Recording config/alerts with fingerprint=25d29ab1e5bbeba543c80be42f0800c439c6b780d4eec335aa12692804c00ee6 I0504 20:37:48.429664 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.055000558s to process 1 records W0504 20:37:49.412971 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 20:37:49.846601 1 tasks_processing.go:74] worker 59 stopped. I0504 20:37:49.846724 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=fc7bb872ae82df6315810b0148121c8cb5d771a29c6aacb631549c3ae652d4b9 I0504 20:37:49.846772 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0504 20:37:49.846806 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=1cc7897a6eeb1ff28d13c838e35163a93ea585049e8d7afebb8e32cdcdd3e543 I0504 20:37:49.846819 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0504 20:37:49.846852 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=2f012608c9b0101559dff76349bbb723bbcaf238909a0a6bc8c1aeab80ec2569 I0504 20:37:49.846875 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0504 20:37:49.846902 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=44f4d97b4908e382b544884052632f94ca40a94aea3bf51aed43025dd7255b4b I0504 20:37:49.846931 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=d4e14b815954ce6fd50d32348f1146ba1f751991a35295fa459601d9f628ce10 I0504 20:37:49.846954 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=56752426977a611049d86303ab2c14a286faf4f183b605bb4fd577f6c4692d14 I0504 20:37:49.846983 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=fffebf06d8db862d6d516f31f2e5a222ab32ae5c382d158d189fc5f360453b2a I0504 20:37:49.847232 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=079ecc1d8a0ceea0cebe7b8448cf0b10216ec2f0d97e3ef52084bb5746ad4005 I0504 20:37:49.847253 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=bf6b1c52af0cf69a9aa9762a440cb45318f9b65fb4080197c3166f92e0d6e9d5 I0504 20:37:49.847262 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0504 20:37:49.847277 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=fbad5c6fba3e97b85439d203ae811e9f2f8431cf9a7fc0cc84104c6a1caf9440 I0504 20:37:49.847288 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0504 20:37:49.847317 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=46a33c09c77892510a91d1016fdfdd4ce5c95cad4bbcb904f80b2c961756ff5c I0504 20:37:49.847327 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0504 20:37:49.847346 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=f60f00ec839af9d34da54ed246a716b5d2d50197d20310fca11be248415dcb3d I0504 20:37:49.847356 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0504 20:37:49.847377 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=4aceeef333405db839fcf399ddc374e875a195d6cf3a198c7dc48d6b83acf3c1 I0504 20:37:49.847480 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=508462e81cfc51583776c4daea2acaaeb5adfd8c99d0e126da91b443d0c988db I0504 20:37:49.847489 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0504 20:37:49.847497 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0504 20:37:49.847517 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0504 20:37:49.847540 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=d6bbfb317eb9b26359910019b7e7ae225873125d8f41fd39060310a3827f1ab1 I0504 20:37:49.847563 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=4087bec0bc8180099b457336b63390d0b348005e5425a8ff7d409207d8e73597 I0504 20:37:49.847573 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0504 20:37:49.847588 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=9a3d3aedb3916eb63fa52e163cb2b910521a0f19a1d26fe281f32c7aaed018aa I0504 20:37:49.847596 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0504 20:37:49.847614 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=f91aaabbc8f41d4afe11ce004537b794b9413dd3dbcaada7a12107c1bd97fe1a I0504 20:37:49.847623 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0504 20:37:49.847637 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=60399377d8590539a1bc8fb6e9c95580d42403817e72487378dad10a8eead95c I0504 20:37:49.847650 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=23885785fc559514cb959c2cdee9f9d0c01394d20fd1ad12cdee0391edcbd440 I0504 20:37:49.847663 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=325b31b3e10f5e796c3fbbb6357961adc23d9eedc6a14633158ff783a85c75f9 I0504 20:37:49.847684 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=9d3d0dec6c50b9cefd428b7baa13a557ffa24ad3a6637d444357b86a5a653748 I0504 20:37:49.847693 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0504 20:37:49.847730 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=fffc7a6c9ab67368588c431e0d9820191664cd81b3201e13ace0e55b4dba397c I0504 20:37:49.847747 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0504 20:37:49.847754 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0504 20:37:49.847761 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.467649015s to process 39 records W0504 20:37:50.412646 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0504 20:37:51.411912 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0504 20:37:52.412252 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0504 20:37:52.412281 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0504 20:37:52.412300 1 tasks_processing.go:74] worker 11 stopped. E0504 20:37:52.412336 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0504 20:37:52.412350 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0504 20:37:52.412360 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.037564521s to process 1 records I0504 20:38:00.216784 1 tasks_processing.go:74] worker 16 stopped. I0504 20:38:00.216827 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0504 20:38:00.216845 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.841722064s to process 1 records I0504 20:38:00.987263 1 tasks_processing.go:74] worker 27 stopped. I0504 20:38:00.987568 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=80e5b054934e37ddf232a6b52185c88ef3149965af63f70b0125fd661e8f707e I0504 20:38:00.987587 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.608049682s to process 1 records E0504 20:38:00.987661 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.613s with: function \"pod_network_connectivity_checks\" failed with an error, function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"machines\" failed with an error, function \"machine_configs\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error" I0504 20:38:00.988772 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "tsdb_status" failed with an error, function "metrics" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "machines" failed with an error, function "machine_configs" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error I0504 20:38:00.988788 1 periodic.go:216] Running workloads gatherer I0504 20:38:00.988804 1 tasks_processing.go:45] number of workers: 2 I0504 20:38:00.988810 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 20:38:00.988814 1 tasks_processing.go:71] worker 1 working on helmchart_info task. I0504 20:38:00.988827 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 20:38:00.988908 1 tasks_processing.go:71] worker 0 working on workload_info task. I0504 20:38:00.994504 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-160.ec2.internal... I0504 20:38:00.994512 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-50.ec2.internal... I0504 20:38:00.994530 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-243.ec2.internal... I0504 20:38:01.015464 1 tasks_processing.go:74] worker 1 stopped. I0504 20:38:01.015482 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 26.635433ms to process 0 records I0504 20:38:05.329797 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 4s I0504 20:38:05.352179 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0504 20:38:05.360995 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (10ms) I0504 20:38:05.369828 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (9ms) I0504 20:38:05.378416 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (9ms) I0504 20:38:05.386844 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (8ms) I0504 20:38:05.395257 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (8ms) I0504 20:38:05.403624 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (8ms) I0504 20:38:05.412281 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (9ms) I0504 20:38:05.421159 1 gather_workloads_info.go:387] No image sha256:3380fecebaeee139c1607fbcd9346cebc92806679568e2244a8905ef0f52aad2 (9ms) I0504 20:38:05.429282 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (8ms) I0504 20:38:05.437676 1 gather_workloads_info.go:387] No image sha256:1eb6a6ee3c66ac15be6bc21bb6015c595c11c15b9dfd21bd9f5ce7f7b990ebaa (8ms) I0504 20:38:05.460688 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (23ms) I0504 20:38:05.566112 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (105ms) I0504 20:38:05.661784 1 gather_workloads_info.go:387] No image sha256:52db05084f2a2e7d7bf4040e4865df88ec4074efbbe59a78d861be2c5ab2aa40 (96ms) I0504 20:38:05.762521 1 gather_workloads_info.go:387] No image sha256:bdeb1c455cffccdb6c90db238e9d371de06340b1d6105d47f66d87293f878ac1 (101ms) I0504 20:38:05.863612 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (101ms) I0504 20:38:05.962860 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (99ms) I0504 20:38:06.060910 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (98ms) I0504 20:38:06.161446 1 gather_workloads_info.go:387] No image sha256:15928a5b803fa00781b408dbc4c3bfa7c5287b18584ba31f817def8a4430c725 (101ms) I0504 20:38:06.263124 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (102ms) I0504 20:38:06.361932 1 gather_workloads_info.go:387] No image sha256:53ff8efdd8f563421ebc1859ab021a3bbc1306f82d4ff5d9c21bc60ec5f57898 (99ms) I0504 20:38:06.461589 1 gather_workloads_info.go:398] Found image sha256:23204877f166455ce8f9de9b91a467eeda576f172d5f96124a68aa986c4775d6 (100ms) I0504 20:38:06.561482 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (100ms) I0504 20:38:06.662615 1 gather_workloads_info.go:387] No image sha256:5d6a2b8d2508c7da71192661e7c0dcf1335f48b6336a20189abffbe79eeff962 (101ms) I0504 20:38:06.761934 1 gather_workloads_info.go:387] No image sha256:0acfe677ed58031f8381cb8a4f2d8917ec4cfa95485e2a46529dffae54c75ace (99ms) I0504 20:38:06.862003 1 gather_workloads_info.go:387] No image sha256:572ebf7c7fab1a518582f3ca46c866730423cce6e8a19f614dc15fd61bf0c65a (100ms) I0504 20:38:06.961271 1 gather_workloads_info.go:387] No image sha256:5940cc34b322325618929b1e0eb3864dae73d54c8f5b474e218dfb28123a61c7 (99ms) I0504 20:38:07.061793 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (100ms) I0504 20:38:07.162229 1 gather_workloads_info.go:387] No image sha256:6c6d412ccf0c3d6a14645a79df6c4658c0f8725d7954cb83c3d2f0116499345a (100ms) I0504 20:38:07.261948 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (100ms) I0504 20:38:07.361660 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (100ms) I0504 20:38:07.462377 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (101ms) I0504 20:38:07.561777 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (99ms) I0504 20:38:07.662230 1 gather_workloads_info.go:398] Found image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (100ms) I0504 20:38:07.761534 1 gather_workloads_info.go:387] No image sha256:0b2e6536a0ca3afc5e64a218087a6287c74b77a359075a385f1949ca5bd51b22 (99ms) I0504 20:38:07.860935 1 gather_workloads_info.go:387] No image sha256:5ff204630794311b0b37fa7b197b933ad85d76a481bf7cdb3bcbada08f0cdcbf (99ms) I0504 20:38:07.960813 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (100ms) I0504 20:38:08.062210 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (101ms) I0504 20:38:08.161749 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (100ms) I0504 20:38:08.262118 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (100ms) I0504 20:38:08.361264 1 gather_workloads_info.go:387] No image sha256:15e1b5f763553e86d0230a751b0783e3b990438fd07cf96eed164bccb72ed564 (99ms) I0504 20:38:08.461596 1 gather_workloads_info.go:387] No image sha256:48f5559439bea030a6423652da448c25fff0c62ba293eb521843c1a8eb89ddb0 (100ms) I0504 20:38:08.561846 1 gather_workloads_info.go:387] No image sha256:17aefdc2f4752c23dc8f0947bb01fc34635822eeb18f2d8a5eddf8211b84d450 (100ms) I0504 20:38:08.662547 1 gather_workloads_info.go:387] No image sha256:9d6d63dce784df5863e6a0ba4c11317a26c9d756d4e62d47759c97190a71cce5 (101ms) I0504 20:38:08.762094 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (100ms) I0504 20:38:08.862632 1 gather_workloads_info.go:387] No image sha256:9ef6dc311b880a6442b7a3a19006ddd955c7da36850bf000f5c9b68fd0862dd4 (101ms) I0504 20:38:08.961807 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (99ms) I0504 20:38:08.961839 1 tasks_processing.go:74] worker 0 stopped. I0504 20:38:08.962118 1 recorder.go:75] Recording config/workload_info with fingerprint=6b4e24dd1ae0058f0c738a2e387f821d3152c3039f1c1c28c6ca674f8fa28f1f I0504 20:38:08.962134 1 gather.go:177] gatherer "workloads" function "workload_info" took 7.972916446s to process 1 records I0504 20:38:08.962148 1 periodic.go:263] Periodic gather workloads completed in 7.973s I0504 20:38:08.962155 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0504 20:38:08.962160 1 periodic.go:216] Running conditional gatherer I0504 20:38:08.967919 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0504 20:38:09.156874 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0504 20:38:10.196719 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0504 20:38:10.202575 1 conditional_gatherer.go:392] cluster version is '4.21.9' I0504 20:38:10.202609 1 tasks_processing.go:45] number of workers: 3 I0504 20:38:10.202630 1 tasks_processing.go:69] worker 2 listening for tasks. I0504 20:38:10.202637 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0504 20:38:10.202644 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 20:38:10.202659 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0504 20:38:10.202662 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 20:38:10.202675 1 tasks_processing.go:74] worker 1 stopped. I0504 20:38:10.202677 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0504 20:38:10.202715 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0504 20:38:10.202731 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.147µs to process 1 records I0504 20:38:10.202816 1 tasks_processing.go:74] worker 2 stopped. I0504 20:38:10.202824 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0504 20:38:10.202834 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.561µs to process 1 records I0504 20:38:10.202954 1 tasks_processing.go:74] worker 0 stopped. I0504 20:38:10.202967 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 285.526µs to process 0 records I0504 20:38:10.202982 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0504 20:38:10.202998 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0504 20:38:10.234220 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=98f37a5f2ea398aa6479bf147656530e6add22f149535f205a5fe6c1a84c41df I0504 20:38:10.234378 1 diskrecorder.go:70] Writing 110 records to /var/lib/insights-operator/insights-2026-05-04-203810.tar.gz I0504 20:38:10.241043 1 diskrecorder.go:51] Wrote 110 records to disk in 6ms I0504 20:38:10.241076 1 periodic.go:285] Gathering cluster info every 2h0m0s I0504 20:38:10.241093 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0504 20:39:17.375010 1 diskrecorder.go:223] Found files to send: insights-2026-05-04-203810.tar.gz I0504 20:39:17.375040 1 insightsuploader.go:150] Checking archives to upload periodically every 15m13.635210162s I0504 20:39:17.375048 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0504 20:39:17.386550 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0504 20:39:17.586949 1 requests.go:87] Successfully reported id=2026-05-04T20:39:17Z x-rh-insights-request-id=76758c769a6f4784b2005cff69391063, wrote=48517 I0504 20:39:17.587005 1 insightsuploader.go:187] Uploaded report successfully in 211.948182ms I0504 20:39:17.587026 1 controller.go:129] Initializing last reported time to 2026-05-04T20:39:17Z I0504 20:39:17.587085 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0504 20:39:17.587095 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0504 20:39:17.587103 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0504 20:39:17.592075 1 controller.go:554] The operator is healthy I0504 20:39:47.388913 1 controller.go:554] The operator is healthy I0504 20:40:19.810553 1 insightsreport.go:137] Pulling report from smart-proxy I0504 20:40:19.810579 1 insightsreport.go:149] Retrieving report I0504 20:40:19.816937 1 requests.go:110] Retrieving report for cluster: 64257270-1164-4add-bc7d-63bae4ee4301 I0504 20:40:19.816949 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/64257270-1164-4add-bc7d-63bae4ee4301/reports I0504 20:40:19.820189 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/64257270-1164-4add-bc7d-63bae4ee4301/reports I0504 20:40:19.991954 1 insightsreport.go:184] Report retrieved I0504 20:40:20.019765 1 insightsreport.go:239] Report retrieved correctly I0504 20:41:47.388345 1 controller.go:554] The operator is healthy I0504 20:42:47.366922 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 20:42:47.371925 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 20:42:47.371943 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 20:42:47.375492 1 secretconfigobserver.go:119] support secret does not exist I0504 20:43:47.388786 1 controller.go:554] The operator is healthy I0504 20:45:47.388652 1 controller.go:554] The operator is healthy I0504 20:47:47.367701 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:47:47.375817 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 20:47:47.379435 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 20:47:47.379449 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 20:47:47.382599 1 secretconfigobserver.go:119] support secret does not exist I0504 20:47:47.386863 1 controller.go:554] The operator is healthy I0504 20:49:47.392360 1 controller.go:554] The operator is healthy I0504 20:51:47.388875 1 controller.go:554] The operator is healthy I0504 20:52:47.383622 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 20:52:47.388552 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 20:52:47.388580 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 20:52:47.391795 1 secretconfigobserver.go:119] support secret does not exist I0504 20:53:47.389110 1 controller.go:554] The operator is healthy I0504 20:54:36.223859 1 insightsuploader.go:146] Nothing to report since 2026-05-04T20:39:17Z I0504 20:55:47.389248 1 controller.go:554] The operator is healthy I0504 20:57:47.365167 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:57:47.388533 1 controller.go:554] The operator is healthy I0504 20:57:47.392284 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 20:57:47.396477 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 20:57:47.396497 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 20:57:47.400117 1 secretconfigobserver.go:119] support secret does not exist I0504 20:59:47.389517 1 controller.go:554] The operator is healthy I0504 21:01:47.389925 1 controller.go:554] The operator is healthy I0504 21:02:47.401109 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 21:02:47.406368 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 21:02:47.406391 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 21:02:47.409923 1 secretconfigobserver.go:119] support secret does not exist I0504 21:03:47.389144 1 controller.go:554] The operator is healthy I0504 21:05:47.389614 1 controller.go:554] The operator is healthy I0504 21:07:47.365927 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 21:07:47.369658 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 21:07:47.387971 1 controller.go:554] The operator is healthy I0504 21:07:47.410775 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0504 21:07:47.414906 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 21:07:47.414927 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0504 21:07:47.418487 1 secretconfigobserver.go:119] support secret does not exist I0504 21:09:47.388984 1 controller.go:554] The operator is healthy I0504 21:09:55.562136 1 insightsuploader.go:146] Nothing to report since 2026-05-04T20:39:17Z