I0424 02:58:26.312110 1 cmd.go:253] Using service-serving-cert provided certificates I0424 02:58:26.312538 1 observer_polling.go:159] Starting file observer I0424 02:58:26.607332 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0424 02:58:26.607530 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0424 02:58:26.608074 1 secure_serving.go:57] Forcing use of http/1.1 only W0424 02:58:26.608094 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0424 02:58:26.608099 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0424 02:58:26.608104 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0424 02:58:26.608107 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0424 02:58:26.608110 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0424 02:58:26.608112 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0424 02:58:26.608206 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0424 02:58:26.612204 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0424 02:58:26.612278 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"74a2fb29-dfa5-405d-a82f-0fa9e3bcc786", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0424 02:58:26.613324 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0424 02:58:26.613349 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0424 02:58:26.613346 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0424 02:58:26.613358 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0424 02:58:26.613371 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0424 02:58:26.613362 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0424 02:58:26.613643 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0424 02:58:26.613969 1 secure_serving.go:213] Serving securely on [::]:8443 I0424 02:58:26.613996 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0424 02:58:26.618244 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0424 02:58:26.618279 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0424 02:58:26.618327 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0424 02:58:26.623579 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 02:58:26.623599 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0424 02:58:26.628461 1 secretconfigobserver.go:119] support secret does not exist I0424 02:58:26.633633 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 02:58:26.638405 1 secretconfigobserver.go:119] support secret does not exist I0424 02:58:26.640827 1 recorder.go:176] Pruning old reports every 5h19m9s, max age is 288h0m0s I0424 02:58:26.646161 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0424 02:58:26.646177 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0424 02:58:26.646185 1 insightsreport.go:296] Starting report retriever I0424 02:58:26.646190 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0424 02:58:26.646177 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0424 02:58:26.646164 1 periodic.go:216] Running clusterconfig gatherer I0424 02:58:26.646229 1 tasks_processing.go:45] number of workers: 64 I0424 02:58:26.646245 1 tasks_processing.go:69] worker 2 listening for tasks. I0424 02:58:26.646249 1 tasks_processing.go:69] worker 5 listening for tasks. I0424 02:58:26.646255 1 tasks_processing.go:69] worker 17 listening for tasks. I0424 02:58:26.646257 1 tasks_processing.go:69] worker 0 listening for tasks. I0424 02:58:26.646263 1 tasks_processing.go:69] worker 6 listening for tasks. I0424 02:58:26.646264 1 tasks_processing.go:71] worker 0 working on schedulers task. I0424 02:58:26.646266 1 tasks_processing.go:69] worker 1 listening for tasks. I0424 02:58:26.646273 1 tasks_processing.go:71] worker 1 working on openshift_logging task. I0424 02:58:26.646275 1 tasks_processing.go:69] worker 7 listening for tasks. I0424 02:58:26.646271 1 tasks_processing.go:69] worker 4 listening for tasks. I0424 02:58:26.646267 1 tasks_processing.go:71] worker 6 working on monitoring_persistent_volumes task. I0424 02:58:26.646287 1 tasks_processing.go:69] worker 9 listening for tasks. I0424 02:58:26.646293 1 tasks_processing.go:69] worker 14 listening for tasks. I0424 02:58:26.646288 1 tasks_processing.go:69] worker 42 listening for tasks. I0424 02:58:26.646298 1 tasks_processing.go:69] worker 15 listening for tasks. I0424 02:58:26.646304 1 tasks_processing.go:69] worker 16 listening for tasks. I0424 02:58:26.646304 1 tasks_processing.go:69] worker 18 listening for tasks. I0424 02:58:26.646263 1 tasks_processing.go:69] worker 3 listening for tasks. I0424 02:58:26.646311 1 tasks_processing.go:69] worker 31 listening for tasks. I0424 02:58:26.646308 1 tasks_processing.go:69] worker 13 listening for tasks. I0424 02:58:26.646320 1 tasks_processing.go:69] worker 32 listening for tasks. I0424 02:58:26.646315 1 tasks_processing.go:69] worker 54 listening for tasks. I0424 02:58:26.646319 1 tasks_processing.go:69] worker 43 listening for tasks. I0424 02:58:26.646330 1 tasks_processing.go:69] worker 12 listening for tasks. I0424 02:58:26.646330 1 tasks_processing.go:69] worker 11 listening for tasks. I0424 02:58:26.646336 1 tasks_processing.go:69] worker 19 listening for tasks. I0424 02:58:26.646342 1 tasks_processing.go:69] worker 37 listening for tasks. I0424 02:58:26.646282 1 tasks_processing.go:69] worker 8 listening for tasks. I0424 02:58:26.646346 1 tasks_processing.go:69] worker 38 listening for tasks. I0424 02:58:26.646350 1 tasks_processing.go:69] worker 52 listening for tasks. I0424 02:58:26.646351 1 tasks_processing.go:69] worker 20 listening for tasks. I0424 02:58:26.646353 1 tasks_processing.go:69] worker 49 listening for tasks. I0424 02:58:26.646352 1 tasks_processing.go:69] worker 10 listening for tasks. I0424 02:58:26.646361 1 tasks_processing.go:69] worker 50 listening for tasks. I0424 02:58:26.646326 1 tasks_processing.go:69] worker 44 listening for tasks. I0424 02:58:26.646360 1 tasks_processing.go:69] worker 60 listening for tasks. I0424 02:58:26.646369 1 tasks_processing.go:69] worker 22 listening for tasks. I0424 02:58:26.646371 1 tasks_processing.go:69] worker 41 listening for tasks. I0424 02:58:26.646375 1 tasks_processing.go:69] worker 56 listening for tasks. I0424 02:58:26.646375 1 tasks_processing.go:69] worker 59 listening for tasks. I0424 02:58:26.646377 1 tasks_processing.go:69] worker 26 listening for tasks. I0424 02:58:26.646381 1 tasks_processing.go:69] worker 29 listening for tasks. I0424 02:58:26.646384 1 tasks_processing.go:71] worker 32 working on machines task. I0424 02:58:26.646386 1 tasks_processing.go:69] worker 61 listening for tasks. I0424 02:58:26.646385 1 tasks_processing.go:69] worker 24 listening for tasks. I0424 02:58:26.646391 1 tasks_processing.go:71] worker 14 working on config_maps task. I0424 02:58:26.646385 1 tasks_processing.go:69] worker 27 listening for tasks. I0424 02:58:26.646398 1 tasks_processing.go:71] worker 4 working on overlapping_namespace_uids task. I0424 02:58:26.646331 1 tasks_processing.go:69] worker 45 listening for tasks. I0424 02:58:26.646397 1 tasks_processing.go:71] worker 52 working on support_secret task. I0424 02:58:26.646407 1 tasks_processing.go:71] worker 50 working on crds task. I0424 02:58:26.646409 1 tasks_processing.go:71] worker 18 working on machine_sets task. I0424 02:58:26.646411 1 tasks_processing.go:71] worker 45 working on ingress task. I0424 02:58:26.646415 1 tasks_processing.go:71] worker 44 working on cost_management_metrics_configs task. I0424 02:58:26.646416 1 tasks_processing.go:71] worker 41 working on storage_cluster task. I0424 02:58:26.646419 1 tasks_processing.go:71] worker 37 working on active_alerts task. I0424 02:58:26.646420 1 tasks_processing.go:71] worker 2 working on aggregated_monitoring_cr_names task. I0424 02:58:26.646454 1 tasks_processing.go:71] worker 31 working on operators task. I0424 02:58:26.646455 1 tasks_processing.go:71] worker 8 working on container_images task. I0424 02:58:26.646460 1 tasks_processing.go:71] worker 38 working on openshift_machine_api_events task. I0424 02:58:26.646395 1 tasks_processing.go:71] worker 10 working on openstack_dataplanenodesets task. I0424 02:58:26.646392 1 tasks_processing.go:71] worker 49 working on infrastructures task. I0424 02:58:26.646415 1 tasks_processing.go:71] worker 5 working on sap_datahubs task. I0424 02:58:26.646336 1 tasks_processing.go:69] worker 46 listening for tasks. I0424 02:58:26.646495 1 tasks_processing.go:71] worker 13 working on openstack_version task. I0424 02:58:26.646499 1 tasks_processing.go:71] worker 19 working on machine_healthchecks task. I0424 02:58:26.646506 1 tasks_processing.go:71] worker 17 working on machine_config_pools task. I0424 02:58:26.646339 1 tasks_processing.go:69] worker 33 listening for tasks. I0424 02:58:26.646517 1 tasks_processing.go:71] worker 56 working on clusterroles task. I0424 02:58:26.646520 1 tasks_processing.go:71] worker 33 working on authentication task. I0424 02:58:26.646345 1 tasks_processing.go:69] worker 48 listening for tasks. I0424 02:58:26.646351 1 tasks_processing.go:69] worker 34 listening for tasks. I0424 02:58:26.646412 1 tasks_processing.go:71] worker 22 working on machine_configs task. I0424 02:58:26.646500 1 tasks_processing.go:71] worker 46 working on container_runtime_configs task. I0424 02:58:26.646559 1 tasks_processing.go:71] worker 48 working on nodenetworkstates task. I0424 02:58:26.646559 1 tasks_processing.go:71] worker 34 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0424 02:58:26.646384 1 tasks_processing.go:71] worker 59 working on sap_config task. I0424 02:58:26.646404 1 tasks_processing.go:71] worker 43 working on node_features task. I0424 02:58:26.646399 1 tasks_processing.go:71] worker 54 working on metrics task. I0424 02:58:26.646357 1 tasks_processing.go:69] worker 39 listening for tasks. I0424 02:58:26.646357 1 tasks_processing.go:69] worker 57 listening for tasks. I0424 02:58:26.646834 1 tasks_processing.go:71] worker 57 working on operators_pods_and_events task. I0424 02:58:26.646358 1 tasks_processing.go:69] worker 35 listening for tasks. I0424 02:58:26.646363 1 tasks_processing.go:69] worker 21 listening for tasks. I0424 02:58:26.646364 1 tasks_processing.go:69] worker 25 listening for tasks. I0424 02:58:26.646363 1 tasks_processing.go:69] worker 40 listening for tasks. I0424 02:58:26.646319 1 tasks_processing.go:69] worker 30 listening for tasks. I0424 02:58:26.646366 1 tasks_processing.go:69] worker 55 listening for tasks. I0424 02:58:26.646368 1 tasks_processing.go:69] worker 58 listening for tasks. I0424 02:58:26.646368 1 tasks_processing.go:69] worker 36 listening for tasks. I0424 02:58:26.646373 1 tasks_processing.go:69] worker 28 listening for tasks. I0424 02:58:26.646379 1 tasks_processing.go:69] worker 23 listening for tasks. I0424 02:58:26.646378 1 tasks_processing.go:69] worker 63 listening for tasks. I0424 02:58:26.646385 1 tasks_processing.go:71] worker 15 working on dvo_metrics task. I0424 02:58:26.646941 1 tasks_processing.go:71] worker 63 working on pod_network_connectivity_checks task. I0424 02:58:26.646385 1 tasks_processing.go:71] worker 26 working on node_logs task. I0424 02:58:26.647510 1 tasks_processing.go:71] worker 55 working on nodenetworkconfigurationpolicies task. I0424 02:58:26.647593 1 tasks_processing.go:71] worker 21 working on ceph_cluster task. I0424 02:58:26.647782 1 tasks_processing.go:71] worker 30 working on cluster_apiserver task. I0424 02:58:26.646387 1 tasks_processing.go:69] worker 62 listening for tasks. I0424 02:58:26.647982 1 tasks_processing.go:71] worker 25 working on tsdb_status task. I0424 02:58:26.648037 1 tasks_processing.go:71] worker 58 working on certificate_signing_requests task. I0424 02:58:26.648101 1 tasks_processing.go:71] worker 40 working on openstack_dataplanedeployments task. I0424 02:58:26.647527 1 tasks_processing.go:71] worker 35 working on proxies task. I0424 02:58:26.648347 1 tasks_processing.go:71] worker 28 working on openstack_controlplanes task. I0424 02:58:26.646390 1 tasks_processing.go:71] worker 20 working on olm_operators task. I0424 02:58:26.646393 1 tasks_processing.go:71] worker 7 working on lokistack task. I0424 02:58:26.646394 1 tasks_processing.go:71] worker 11 working on image_pruners task. I0424 02:58:26.648050 1 tasks_processing.go:71] worker 62 working on pdbs task. I0424 02:58:26.646395 1 tasks_processing.go:71] worker 24 working on feature_gates task. I0424 02:58:26.646394 1 tasks_processing.go:71] worker 61 working on mutating_webhook_configurations task. I0424 02:58:26.646394 1 tasks_processing.go:71] worker 42 working on validating_webhook_configurations task. I0424 02:58:26.646400 1 tasks_processing.go:71] worker 3 working on machine_autoscalers task. I0424 02:58:26.646402 1 tasks_processing.go:71] worker 60 working on ingress_certificates task. I0424 02:58:26.646405 1 tasks_processing.go:71] worker 16 working on image task. I0424 02:58:26.646404 1 tasks_processing.go:71] worker 27 working on version task. I0424 02:58:26.646408 1 tasks_processing.go:71] worker 12 working on install_plans task. I0424 02:58:26.646409 1 tasks_processing.go:71] worker 29 working on qemu_kubevirt_launcher_logs task. I0424 02:58:26.646340 1 tasks_processing.go:69] worker 47 listening for tasks. I0424 02:58:26.646347 1 tasks_processing.go:69] worker 51 listening for tasks. I0424 02:58:26.646823 1 tasks_processing.go:71] worker 39 working on service_accounts task. I0424 02:58:26.646356 1 tasks_processing.go:69] worker 53 listening for tasks. I0424 02:58:26.646389 1 tasks_processing.go:71] worker 9 working on silenced_alerts task. I0424 02:58:26.648280 1 tasks_processing.go:71] worker 23 working on nodes task. I0424 02:58:26.648914 1 tasks_processing.go:71] worker 36 working on image_registries task. I0424 02:58:26.650267 1 tasks_processing.go:71] worker 47 working on storage_classes task. I0424 02:58:26.650292 1 tasks_processing.go:71] worker 51 working on jaegers task. I0424 02:58:26.650389 1 tasks_processing.go:71] worker 17 working on oauths task. I0424 02:58:26.650397 1 tasks_processing.go:71] worker 53 working on networks task. I0424 02:58:26.650510 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 3.873032ms to process 0 records I0424 02:58:26.650654 1 tasks_processing.go:71] worker 1 working on sap_pods task. I0424 02:58:26.650661 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 4.370578ms to process 0 records E0424 02:58:26.652338 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.255.38:9091: connect: connection refused I0424 02:58:26.652353 1 tasks_processing.go:74] worker 25 stopped. E0424 02:58:26.652359 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.255.38:9091: connect: connection refused I0424 02:58:26.652368 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 4.319527ms to process 0 records E0424 02:58:26.652385 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.255.38:9091: connect: connection refused I0424 02:58:26.652398 1 tasks_processing.go:74] worker 54 stopped. E0424 02:58:26.652412 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.255.38:9091: connect: connection refused I0424 02:58:26.652433 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 5.657595ms to process 0 records I0424 02:58:26.652841 1 tasks_processing.go:74] worker 48 stopped. I0424 02:58:26.652852 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 6.27446ms to process 0 records I0424 02:58:26.652922 1 tasks_processing.go:74] worker 32 stopped. E0424 02:58:26.652950 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0424 02:58:26.652962 1 gather.go:177] gatherer "clusterconfig" function "machines" took 6.526706ms to process 0 records I0424 02:58:26.653204 1 tasks_processing.go:74] worker 59 stopped. I0424 02:58:26.653216 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 6.58001ms to process 0 records I0424 02:58:26.653425 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0424 02:58:26.653439 1 controller.go:368] The initial operator extension status is healthy I0424 02:58:26.653446 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0424 02:58:26.653450 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0424 02:58:26.653454 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0424 02:58:26.653467 1 controller.go:531] The operator is still being initialized I0424 02:58:26.653473 1 controller.go:554] The operator is healthy I0424 02:58:26.653528 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0424 02:58:26.653538 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0424 02:58:26.653639 1 operator.go:328] started I0424 02:58:26.653662 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0424 02:58:26.655150 1 tasks_processing.go:74] worker 0 stopped. I0424 02:58:26.655238 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=96a653efad9478a2aac1f200d71cf96fa6e785b8cf032d6836afd81eecbba6e8 I0424 02:58:26.655251 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 8.871925ms to process 1 records I0424 02:58:26.655717 1 tasks_processing.go:74] worker 63 stopped. E0424 02:58:26.655726 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0424 02:58:26.655732 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 8.759404ms to process 0 records I0424 02:58:26.658843 1 tasks_processing.go:74] worker 46 stopped. I0424 02:58:26.658852 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 12.282675ms to process 0 records I0424 02:58:26.671718 1 tasks_processing.go:74] worker 55 stopped. E0424 02:58:26.671720 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0424 02:58:26.671742 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 24.143791ms to process 0 records I0424 02:58:26.671755 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 25.307781ms to process 0 records I0424 02:58:26.671760 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 24.996917ms to process 0 records I0424 02:58:26.671766 1 tasks_processing.go:74] worker 43 stopped. I0424 02:58:26.671777 1 tasks_processing.go:74] worker 44 stopped. I0424 02:58:26.678259 1 tasks_processing.go:74] worker 41 stopped. I0424 02:58:26.678273 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 31.832855ms to process 0 records I0424 02:58:26.678282 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 31.84611ms to process 0 records I0424 02:58:26.678289 1 tasks_processing.go:74] worker 18 stopped. I0424 02:58:26.678305 1 tasks_processing.go:74] worker 40 stopped. I0424 02:58:26.678315 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 30.164765ms to process 0 records I0424 02:58:26.678459 1 tasks_processing.go:74] worker 30 stopped. I0424 02:58:26.678622 1 recorder.go:75] Recording config/apiserver with fingerprint=87a51b6a6f030f3d890619ed3c7476a60de2f7601b97d4fddfb9ceea418ff4e4 I0424 02:58:26.678636 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 30.619042ms to process 1 records I0424 02:58:26.679472 1 tasks_processing.go:74] worker 10 stopped. I0424 02:58:26.679481 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 32.99517ms to process 0 records I0424 02:58:26.679487 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 32.973232ms to process 0 records I0424 02:58:26.679494 1 tasks_processing.go:74] worker 13 stopped. I0424 02:58:26.679656 1 tasks_processing.go:74] worker 49 stopped. I0424 02:58:26.680161 1 recorder.go:75] Recording config/infrastructure with fingerprint=c9a1dc7dabe9de6a99dbc010294a0340b432e50720c9cb960c6bfd7b1ccb822a I0424 02:58:26.680179 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 33.165337ms to process 1 records I0424 02:58:26.680189 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 32.258758ms to process 0 records I0424 02:58:26.680195 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 31.516304ms to process 0 records I0424 02:58:26.680200 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 30.892492ms to process 0 records I0424 02:58:26.680205 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 33.694218ms to process 0 records I0424 02:58:26.680211 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 33.624961ms to process 0 records I0424 02:58:26.680215 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 31.289292ms to process 0 records I0424 02:58:26.680224 1 tasks_processing.go:74] worker 7 stopped. I0424 02:58:26.680229 1 tasks_processing.go:74] worker 21 stopped. I0424 02:58:26.680235 1 tasks_processing.go:74] worker 28 stopped. I0424 02:58:26.680240 1 tasks_processing.go:74] worker 3 stopped. I0424 02:58:26.680243 1 tasks_processing.go:74] worker 6 stopped. I0424 02:58:26.680245 1 tasks_processing.go:74] worker 38 stopped. I0424 02:58:26.680293 1 tasks_processing.go:74] worker 52 stopped. E0424 02:58:26.680303 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0424 02:58:26.680309 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 33.88186ms to process 0 records I0424 02:58:26.680331 1 tasks_processing.go:74] worker 5 stopped. I0424 02:58:26.680338 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 33.836841ms to process 0 records I0424 02:58:26.680437 1 tasks_processing.go:74] worker 35 stopped. I0424 02:58:26.680513 1 recorder.go:75] Recording config/proxy with fingerprint=f1bba39a4ab4ac070644ff6b8653440ebf3d1881ec0b6bacbadd58cf96547cc9 I0424 02:58:26.680537 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 32.108182ms to process 1 records I0424 02:58:26.682059 1 tasks_processing.go:74] worker 45 stopped. I0424 02:58:26.682229 1 recorder.go:75] Recording config/ingress with fingerprint=5413ab07ed16c0b80aeb418388816e29d5f34899d01de0299504f5942b8bf7b5 I0424 02:58:26.682240 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 35.633571ms to process 1 records I0424 02:58:26.682397 1 tasks_processing.go:74] worker 11 stopped. I0424 02:58:26.682631 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=8de64488ae59d3d5f2a1d43a23155463c0c1d4080c696468c58f41de366c4387 I0424 02:58:26.682645 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 33.402811ms to process 1 records I0424 02:58:26.688153 1 tasks_processing.go:74] worker 26 stopped. I0424 02:58:26.688176 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 40.711715ms to process 0 records I0424 02:58:26.688238 1 tasks_processing.go:74] worker 20 stopped. I0424 02:58:26.688273 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 39.574571ms to process 0 records I0424 02:58:26.688613 1 tasks_processing.go:74] worker 33 stopped. I0424 02:58:26.689098 1 recorder.go:75] Recording config/authentication with fingerprint=6a69090f22c97bc5b42880aededf5ae295891ad27b0c458fbab4bf188ec7bf03 I0424 02:58:26.689126 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 42.083873ms to process 1 records I0424 02:58:26.689159 1 tasks_processing.go:74] worker 16 stopped. I0424 02:58:26.689465 1 recorder.go:75] Recording config/image with fingerprint=e918e262844920ed9d5ac3f344910471d312cad3499703c870ef00135dafd9c4 I0424 02:58:26.689503 1 gather.go:177] gatherer "clusterconfig" function "image" took 40.049128ms to process 1 records I0424 02:58:26.689637 1 tasks_processing.go:74] worker 61 stopped. I0424 02:58:26.690999 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=096baebc1525aa2cda6667daf2147715d9729a28741777dd3791c2e011232394 I0424 02:58:26.691122 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=3c5e49cf7ab7b50e9e036d4995a9fc57a548b6c35b3034e96372610c74f4ccc5 I0424 02:58:26.692311 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=706c17d37e82f8d890f5a302278118dbb8d886e302fe3f0f92384968da9b019a I0424 02:58:26.692337 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 40.487723ms to process 3 records E0424 02:58:26.692351 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0424 02:58:26.692360 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 43.858262ms to process 0 records I0424 02:58:26.692375 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 40.812315ms to process 0 records I0424 02:58:26.692414 1 tasks_processing.go:74] worker 19 stopped. I0424 02:58:26.692455 1 tasks_processing.go:74] worker 51 stopped. I0424 02:58:26.692535 1 recorder.go:75] Recording config/featuregate with fingerprint=fae2ae84154981f88dedc5ae355f8a0488f39a3cf4dc4cfaf7da195b4fe5af65 I0424 02:58:26.692550 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 42.785603ms to process 1 records I0424 02:58:26.692552 1 tasks_processing.go:74] worker 24 stopped. I0424 02:58:26.692556 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 41.286246ms to process 0 records I0424 02:58:26.692566 1 tasks_processing.go:74] worker 1 stopped. I0424 02:58:26.692656 1 tasks_processing.go:74] worker 47 stopped. I0424 02:58:26.692680 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=cc4bf3ce57ff65fb8e0a4cfb422ae2470d368d75cdd402ef98735f1eceaf73a7 I0424 02:58:26.692808 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=ea81484c18416d53ce866ce7557370f1a58bbec474e96910c5fad87aaf234939 I0424 02:58:26.692827 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 41.806222ms to process 2 records I0424 02:58:26.692964 1 tasks_processing.go:74] worker 53 stopped. I0424 02:58:26.693089 1 recorder.go:75] Recording config/network with fingerprint=a403e694fbfad318e4ceaf6721785218649bb75f64d538a0246c326099a26c19 I0424 02:58:26.693115 1 gather.go:177] gatherer "clusterconfig" function "networks" took 41.964789ms to process 1 records E0424 02:58:26.693128 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0424 02:58:26.693145 1 tasks_processing.go:74] worker 22 stopped. I0424 02:58:26.693154 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0424 02:58:26.693165 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 46.122121ms to process 1 records I0424 02:58:26.693186 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0424 02:58:26.693193 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 46.523266ms to process 1 records I0424 02:58:26.693204 1 tasks_processing.go:74] worker 4 stopped. I0424 02:58:26.693618 1 tasks_processing.go:74] worker 62 stopped. I0424 02:58:26.693664 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0424 02:58:26.693675 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0424 02:58:26.693680 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0424 02:58:26.693684 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0424 02:58:26.693689 1 controller.go:254] Source scaController *sca.Controller is not ready I0424 02:58:26.693709 1 controller.go:531] The operator is still being initialized I0424 02:58:26.693721 1 controller.go:554] The operator is healthy I0424 02:58:26.693743 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=d902b3c13f8e153ed17666c0683771df2155a017ca098b3e305ad9eabaa6d1dd I0424 02:58:26.693764 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=c4f1bf1f58074d7ede848e2ecf9b058591d7042a7a1f763626907fb51f3d3961 I0424 02:58:26.693783 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=3730763b2832b92c0251529a3843bb90c3cac343fdeb7eb635ba44a21376c7c5 I0424 02:58:26.693807 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=a015fbaf61ad041a6dd2d5035ca8a02489be1ff67708e020b57f1fd7d8cfebb5 I0424 02:58:26.693835 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=e6f9c1e8b00a174a05a67cf23886815f7ec9ecd749e9ccfca0dfef8cdad6dcf8 I0424 02:58:26.693859 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=04e8d92ffdd8c7e9523e44bd90c9d77fa8737facefaeaa93417e49d1d6f33768 I0424 02:58:26.693884 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=1d0578e65c497cba6f7a615c17afc05e749244d3b982ee75cfdf9ef2feb27959 I0424 02:58:26.693911 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=40a3b2130d3f944a65012ff2e7bb396878f484c6e0dc5d20feba7095159b6d13 I0424 02:58:26.693955 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=b61cc0abf4c640b9f7e37d576501b21f527ed52f4115e975f8644a629e6a84ed I0424 02:58:26.693976 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=02a294d6a83fd35f8000bf412322bde36d133bf933ed61e293fd1f73698032c4 I0424 02:58:26.694000 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=e565844f5b74461ef49dc503ba9e7ab992d8c57383d4bbdb50c11bbe148a5bd0 I0424 02:58:26.694008 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 44.621746ms to process 11 records W0424 02:58:26.694084 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0424 02:58:26.694180 1 tasks_processing.go:74] worker 42 stopped. I0424 02:58:26.694320 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=86682d7fafdd99ddb012647fffbc44098b19e661b12658c6958b76f06ad31f4b I0424 02:58:26.694470 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=a7a6248ccedcd48e8b7c4092a836392072f24ad29ce48597cd0fe8957c749e88 I0424 02:58:26.694478 1 gather_logs.go:145] no pods in namespace were found I0424 02:58:26.695609 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=10c2296a3f1c54922dbf721c4f310f4704640705bd23bd271f88b79d53d9c83e I0424 02:58:26.695725 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=d3d56d9cdc57c0ae574862e56869d5ea6107c7e36d55b47867b7f060eb6fcdb1 I0424 02:58:26.695781 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=bdb6140feed3e25c3e53e8711030d595be5abb711c644a8cdbab964c6c36b65c I0424 02:58:26.695871 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=523685af50162a71026ef7735256d5389488b04c1da3e154f950d28ce8c48685 I0424 02:58:26.696668 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=b893dd2b98f3cb08579fb6c5026d786ca08a685709f1809c400953d3c149bf26 I0424 02:58:26.696799 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=31a0666d783785ae4dfb5c52b58598e27bd16a6a184a68bce6debe61103d775d I0424 02:58:26.696896 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=f2fa221e09970907fc910fc89ad5513a54630db304978eaf1537f4463aa89c06 I0424 02:58:26.696983 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=014d1ffed780fd200f4d827bd14268eb298cd3b212cda5c4060c585c443a0920 I0424 02:58:26.697078 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=159a43974a15a3b58e7d4594ea22323f0c14c5a19d5aa30e25fb9520b847f77c I0424 02:58:26.697140 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=46ba1a2b431f8d6baf15bef7c98c6d7fd7796397bfb3ff37d19b5a0f1cddaea7 I0424 02:58:26.697199 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=62cf181be135945fc418e6d9bcda8b2018ef2f41c297a2dafd982dccb07ca1ae I0424 02:58:26.697236 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 45.145904ms to process 13 records I0424 02:58:26.697265 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 45.369051ms to process 0 records I0424 02:58:26.697297 1 tasks_processing.go:74] worker 29 stopped. I0424 02:58:26.697605 1 tasks_processing.go:74] worker 58 stopped. I0424 02:58:26.697660 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 49.505321ms to process 0 records I0424 02:58:26.698610 1 tasks_processing.go:74] worker 57 stopped. I0424 02:58:26.698655 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 51.763251ms to process 0 records I0424 02:58:26.700445 1 tasks_processing.go:74] worker 50 stopped. I0424 02:58:26.701076 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=016467c3b4f6272770449a9b4c5dc30d10c92465d3d0f728450163e01fe7c239 I0424 02:58:26.701340 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=a50a5e80e28cb7c7b3ffbc28b1cf83cd7134492d6d24bcbcfb9a95a52d7a8c6a I0424 02:58:26.701349 1 gather.go:177] gatherer "clusterconfig" function "crds" took 54.02683ms to process 2 records I0424 02:58:26.701440 1 tasks_processing.go:74] worker 23 stopped. I0424 02:58:26.701684 1 recorder.go:75] Recording config/node/ip-10-0-0-104.ec2.internal with fingerprint=2d9f5c992b9efa88b134a9719113909581fb0baf97872c38946a1bcca1bf469e I0424 02:58:26.701753 1 recorder.go:75] Recording config/node/ip-10-0-1-139.ec2.internal with fingerprint=bfb3c77c6ef830983299f2745920e96fd3ccdf6279875292f710a3e3710f6011 I0424 02:58:26.701813 1 recorder.go:75] Recording config/node/ip-10-0-2-137.ec2.internal with fingerprint=74362a695de2df5c7d7ec76c89ce1097367427b659dee08e3ef102207139dc78 I0424 02:58:26.701823 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 51.318248ms to process 3 records I0424 02:58:26.701843 1 tasks_processing.go:74] worker 27 stopped. I0424 02:58:26.702307 1 recorder.go:75] Recording config/version with fingerprint=4c6aad127584ae85071ceab5f1141442d7507dd90da4820274ffed99eaa25a0c I0424 02:58:26.702321 1 recorder.go:75] Recording config/id with fingerprint=0caec8f81b7acb5ee3d111b82e8409c2646ff2af5fa9cfb512c8975d0891a948 I0424 02:58:26.702328 1 gather.go:177] gatherer "clusterconfig" function "version" took 52.720365ms to process 2 records I0424 02:58:26.702414 1 tasks_processing.go:74] worker 56 stopped. I0424 02:58:26.702543 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=db97be1d84bd74f25efabc1b60850df8deb124a342777d56e71915fd4f2bdcaf I0424 02:58:26.702695 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=b659de824e1056043b43ccc1b58e952c0212897725e63b14e34abec6e7a42dab I0424 02:58:26.702709 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 55.545547ms to process 2 records I0424 02:58:26.702807 1 tasks_processing.go:74] worker 17 stopped. I0424 02:58:26.702888 1 recorder.go:75] Recording config/oauth with fingerprint=6eb0dc0e52fa358d9613e161ff25e1419513216d259c6af39acffad13df847a6 I0424 02:58:26.702896 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 52.175676ms to process 1 records I0424 02:58:26.702957 1 tasks_processing.go:74] worker 36 stopped. I0424 02:58:26.703232 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=cc2c6157d0e209d06e66df978a511ee7e53635e61edd575080d1b47b790d3fda I0424 02:58:26.703243 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 53.341901ms to process 1 records I0424 02:58:26.707337 1 tasks_processing.go:74] worker 2 stopped. I0424 02:58:26.707354 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 60.901038ms to process 0 records I0424 02:58:26.713397 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0424 02:58:26.713417 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0424 02:58:26.713422 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0424 02:58:26.717233 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0424 02:58:26.718367 1 base_controller.go:82] Caches are synced for ConfigController I0424 02:58:26.718382 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0424 02:58:26.723392 1 configmapobserver.go:84] configmaps "insights-config" not found I0424 02:58:26.724717 1 tasks_processing.go:74] worker 14 stopped. E0424 02:58:26.724730 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0424 02:58:26.724736 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0424 02:58:26.724740 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0424 02:58:26.724748 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0424 02:58:26.724784 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0424 02:58:26.724790 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0424 02:58:26.724795 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0424 02:58:26.724805 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0424 02:58:26.724884 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0424 02:58:26.724891 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0424 02:58:26.724897 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 78.311376ms to process 7 records I0424 02:58:26.725174 1 tasks_processing.go:74] worker 60 stopped. E0424 02:58:26.725187 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0424 02:58:26.725194 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2ps8kd9n54vkadsmhv5dl8r11de9333h-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2ps8kd9n54vkadsmhv5dl8r11de9333h-primary-cert-bundle-secret" not found I0424 02:58:26.725241 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=edcaf01304285f190477b9a8377b680c6723b06cdb8229e2c58c4386f1f44a04 I0424 02:58:26.725253 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 76.065568ms to process 1 records I0424 02:58:26.725513 1 tasks_processing.go:74] worker 8 stopped. I0424 02:58:26.726849 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-66dbbb6468-dkjb8 with fingerprint=da4b907a212225f17ad077cef7bf234a79718d45dfbc0e71aed6e8ed76993301 I0424 02:58:26.726926 1 recorder.go:75] Recording config/running_containers with fingerprint=44a5fd1a48944c8de89f678cd41a236b9e4e3a945b88cc7f38d0a7dc275dab46 I0424 02:58:26.726956 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 79.044627ms to process 2 records I0424 02:58:26.727489 1 tasks_processing.go:74] worker 34 stopped. I0424 02:58:26.727506 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 80.91853ms to process 0 records I0424 02:58:26.754242 1 base_controller.go:82] Caches are synced for LoggingSyncer I0424 02:58:26.754259 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0424 02:58:26.760001 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0424 02:58:26.768575 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0424 02:58:26.792126 1 cluster_transfer.go:102] no available accepted cluster transfer I0424 02:58:26.792147 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0424 02:58:26.877417 1 sca.go:266] etc-pki-entitlement secret successfully created I0424 02:58:26.877497 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0424 02:58:27.688829 1 tasks_processing.go:74] worker 9 stopped. I0424 02:58:27.688852 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0424 02:58:27.688865 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.039606685s to process 1 records I0424 02:58:27.689885 1 tasks_processing.go:74] worker 37 stopped. I0424 02:58:27.689944 1 recorder.go:75] Recording config/alerts with fingerprint=bd012df6c544f31f2613a24a7c82ce58d7369c46b73d8f0b38caa6fe92f55d79 I0424 02:58:27.689958 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.043446835s to process 1 records W0424 02:58:27.691442 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0424 02:58:28.692292 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0424 02:58:29.122058 1 tasks_processing.go:74] worker 31 stopped. I0424 02:58:29.122173 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=cfc97e88d9e9f4da38326a4732929c23bb27569165bfc51928878d81f1b8b2a8 I0424 02:58:29.122218 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0424 02:58:29.122247 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=325cc1f44ad08389600f091f34ed066fa0d87f483dac3c9ce8c276083bf722d5 I0424 02:58:29.122257 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0424 02:58:29.122278 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=5bc94b5a265001270c5e128d64c3c9ee8d2dccc30be6f157c21ad07ab0939ae3 I0424 02:58:29.122294 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0424 02:58:29.122314 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=ac832acff4333d4e9c447b482fc35dc52319ebad47340fd8affb8b5d16ca2fe8 I0424 02:58:29.122340 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=b7a3dc3ee64fefe8fb537c14da0827ffc52109cf7f8a156014e25d228d82ffa9 I0424 02:58:29.122366 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=1cedad4a0788fd10ab3eb17886e5d74ea67ebd4d4a8a06badf474034719b83fb I0424 02:58:29.122399 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=973b525d04fce43a36f466aaec400b583cc4e6290ba7bc4812f1beece516c4b7 I0424 02:58:29.122716 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=525a324409edc5e0630b1dd1d3d69a7603dcb62f3ba3c862a7d709a3ecd4feb2 I0424 02:58:29.122738 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=833a4ea7a448ba2253f2311d7b2f76b6fbdacdca50adfc46f5b774d86f786cc4 I0424 02:58:29.122747 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0424 02:58:29.122763 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=40f4e30dfbe168d22c8b11d90059dee8dd3d186d76a8a0e4f675aa7980bb262d I0424 02:58:29.122773 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0424 02:58:29.122790 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=2a5616401286f7f23773a733651ff54d20e2835b8706ca50bfe05da8b4a81be5 I0424 02:58:29.122799 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0424 02:58:29.122818 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=70dede555a2b9595c2f05c8c63f6186b2ca75b90fe45ab5025284b9920d92d2a I0424 02:58:29.122828 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0424 02:58:29.122849 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=552b9694d4fc4e35ee4f57b162426d9ab4314bb46b806b437ab16dafe4d59bb6 I0424 02:58:29.122974 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=fc40090f697774cec0b0f4faa528665e4c768c7b6c70f15f52b698ce0ac9b5e5 I0424 02:58:29.122984 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0424 02:58:29.122991 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0424 02:58:29.123012 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0424 02:58:29.123038 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=d75140640a5b99e0a8bea0535272ea67a705fec82161fd5aeffd5eb8e2c5c16a I0424 02:58:29.123061 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=3cf4f03d6797c0c139a83152da03155e7de0d7f972de58cb58f3bea8311c1812 I0424 02:58:29.123072 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0424 02:58:29.123087 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=44518f4a6dab6043b220c021d69360fe1bb823850562e0ca8454b8e25dad73aa I0424 02:58:29.123097 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0424 02:58:29.123114 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=b1d67faf9a801f38186af733c205f300d61f16dbc1bc46e09f7bb50127fad45b I0424 02:58:29.123123 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0424 02:58:29.123137 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=9b21015f8383548e998c5f7e931f5bfc102a9fec953a0ded3c495bfb8d1111db I0424 02:58:29.123154 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=4e34d7c2d43efc761e09128f6508ef0e5b840dd94a67446468579606552c3f39 I0424 02:58:29.123166 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=d1a22b246c80f55691d17834d928af52ddbca9455f526cc5354402bf678743a4 I0424 02:58:29.123189 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=d8379a908dd5175b111dca6bfe95d20e078641b15d5bc4041597910010f1f54f I0424 02:58:29.123199 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0424 02:58:29.123225 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=a19135dc90d6c8332281dce4f0638173632964aba16e706ed1862cba0fdc908a I0424 02:58:29.123242 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0424 02:58:29.123251 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0424 02:58:29.123257 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.475585676s to process 39 records W0424 02:58:29.691392 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0424 02:58:30.691506 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0424 02:58:31.690536 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0424 02:58:31.690564 1 tasks_processing.go:74] worker 15 stopped. E0424 02:58:31.690586 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0424 02:58:31.690605 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0424 02:58:31.690617 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.043688851s to process 1 records I0424 02:58:39.507056 1 tasks_processing.go:74] worker 12 stopped. I0424 02:58:39.507099 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0424 02:58:39.507113 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.85792692s to process 1 records I0424 02:58:40.255701 1 tasks_processing.go:74] worker 39 stopped. I0424 02:58:40.256020 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=80e5b054934e37ddf232a6b52185c88ef3149965af63f70b0125fd661e8f707e I0424 02:58:40.256041 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.606551319s to process 1 records E0424 02:58:40.256111 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.609s with: function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0424 02:58:40.257222 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "tsdb_status" failed with an error, function "metrics" failed with an error, function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "machine_healthchecks" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0424 02:58:40.257240 1 periodic.go:216] Running workloads gatherer I0424 02:58:40.257257 1 tasks_processing.go:45] number of workers: 2 I0424 02:58:40.257266 1 tasks_processing.go:69] worker 1 listening for tasks. I0424 02:58:40.257271 1 tasks_processing.go:71] worker 1 working on workload_info task. I0424 02:58:40.257280 1 tasks_processing.go:69] worker 0 listening for tasks. I0424 02:58:40.257290 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0424 02:58:40.263300 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-104.ec2.internal... I0424 02:58:40.263308 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-137.ec2.internal... I0424 02:58:40.263332 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-139.ec2.internal... I0424 02:58:40.289102 1 tasks_processing.go:74] worker 0 stopped. I0424 02:58:40.289124 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 31.800172ms to process 0 records I0424 02:58:44.732503 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 4s I0424 02:58:44.757002 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0424 02:58:44.765524 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (10ms) I0424 02:58:44.774376 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (9ms) I0424 02:58:44.783258 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (9ms) I0424 02:58:44.791903 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (9ms) I0424 02:58:44.800485 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (9ms) I0424 02:58:44.809266 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (9ms) I0424 02:58:44.817342 1 gather_workloads_info.go:387] No image sha256:17aefdc2f4752c23dc8f0947bb01fc34635822eeb18f2d8a5eddf8211b84d450 (8ms) I0424 02:58:44.825723 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (8ms) I0424 02:58:44.833757 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (8ms) I0424 02:58:44.841917 1 gather_workloads_info.go:398] Found image sha256:23204877f166455ce8f9de9b91a467eeda576f172d5f96124a68aa986c4775d6 (8ms) I0424 02:58:44.867126 1 gather_workloads_info.go:387] No image sha256:6c6d412ccf0c3d6a14645a79df6c4658c0f8725d7954cb83c3d2f0116499345a (25ms) I0424 02:58:44.966212 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (99ms) I0424 02:58:45.066405 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (100ms) I0424 02:58:45.169386 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (103ms) I0424 02:58:45.266398 1 gather_workloads_info.go:387] No image sha256:53ff8efdd8f563421ebc1859ab021a3bbc1306f82d4ff5d9c21bc60ec5f57898 (97ms) I0424 02:58:45.366745 1 gather_workloads_info.go:387] No image sha256:9d6d63dce784df5863e6a0ba4c11317a26c9d756d4e62d47759c97190a71cce5 (100ms) I0424 02:58:45.465670 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (99ms) I0424 02:58:45.568729 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (103ms) I0424 02:58:45.666502 1 gather_workloads_info.go:387] No image sha256:b958e39c1643d98648b20ff8b6ec38a51c9c6606f2eecff4044d41ac993e1878 (98ms) I0424 02:58:45.767155 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (101ms) I0424 02:58:45.866315 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (99ms) I0424 02:58:45.966019 1 gather_workloads_info.go:387] No image sha256:15928a5b803fa00781b408dbc4c3bfa7c5287b18584ba31f817def8a4430c725 (100ms) I0424 02:58:46.066137 1 gather_workloads_info.go:387] No image sha256:572ebf7c7fab1a518582f3ca46c866730423cce6e8a19f614dc15fd61bf0c65a (100ms) I0424 02:58:46.167353 1 gather_workloads_info.go:387] No image sha256:15e1b5f763553e86d0230a751b0783e3b990438fd07cf96eed164bccb72ed564 (101ms) I0424 02:58:46.267290 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (100ms) I0424 02:58:46.370313 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (103ms) I0424 02:58:46.469162 1 gather_workloads_info.go:387] No image sha256:5ff204630794311b0b37fa7b197b933ad85d76a481bf7cdb3bcbada08f0cdcbf (99ms) I0424 02:58:46.568202 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (99ms) I0424 02:58:46.670326 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (102ms) I0424 02:58:46.766400 1 gather_workloads_info.go:387] No image sha256:3380fecebaeee139c1607fbcd9346cebc92806679568e2244a8905ef0f52aad2 (96ms) I0424 02:58:46.867580 1 gather_workloads_info.go:387] No image sha256:10393129e11d4b44a1a629bdafcf2f45c998616319122d2d7723dd3efd9a3cfe (101ms) I0424 02:58:46.966256 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (99ms) I0424 02:58:47.066511 1 gather_workloads_info.go:387] No image sha256:bdeb1c455cffccdb6c90db238e9d371de06340b1d6105d47f66d87293f878ac1 (100ms) I0424 02:58:47.167066 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (101ms) I0424 02:58:47.267037 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (100ms) I0424 02:58:47.366512 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (99ms) I0424 02:58:47.465865 1 gather_workloads_info.go:387] No image sha256:48f5559439bea030a6423652da448c25fff0c62ba293eb521843c1a8eb89ddb0 (99ms) I0424 02:58:47.569423 1 gather_workloads_info.go:387] No image sha256:0b2e6536a0ca3afc5e64a218087a6287c74b77a359075a385f1949ca5bd51b22 (104ms) I0424 02:58:47.666621 1 gather_workloads_info.go:387] No image sha256:5d6a2b8d2508c7da71192661e7c0dcf1335f48b6336a20189abffbe79eeff962 (97ms) I0424 02:58:47.768024 1 gather_workloads_info.go:387] No image sha256:5940cc34b322325618929b1e0eb3864dae73d54c8f5b474e218dfb28123a61c7 (101ms) I0424 02:58:47.866730 1 gather_workloads_info.go:387] No image sha256:0acfe677ed58031f8381cb8a4f2d8917ec4cfa95485e2a46529dffae54c75ace (99ms) I0424 02:58:47.968873 1 gather_workloads_info.go:387] No image sha256:0092742d8c2f3e9241f8f87d6f1da2a063627c140c491dc9c5e97526f760101a (102ms) I0424 02:58:48.065840 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (97ms) I0424 02:58:48.171114 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (105ms) I0424 02:58:48.266166 1 gather_workloads_info.go:387] No image sha256:1eb6a6ee3c66ac15be6bc21bb6015c595c11c15b9dfd21bd9f5ce7f7b990ebaa (95ms) I0424 02:58:48.374730 1 gather_workloads_info.go:398] Found image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (109ms) I0424 02:58:48.465992 1 gather_workloads_info.go:387] No image sha256:52db05084f2a2e7d7bf4040e4865df88ec4074efbbe59a78d861be2c5ab2aa40 (91ms) I0424 02:58:48.570762 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (105ms) I0424 02:58:48.665969 1 gather_workloads_info.go:387] No image sha256:9ef6dc311b880a6442b7a3a19006ddd955c7da36850bf000f5c9b68fd0862dd4 (95ms) I0424 02:58:48.666000 1 tasks_processing.go:74] worker 1 stopped. I0424 02:58:48.666319 1 recorder.go:75] Recording config/workload_info with fingerprint=b43ebcf359e55fb2e26b9015e80996759eafef00ff19e19d1bfd3e40940e08cf I0424 02:58:48.666335 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.408722333s to process 1 records I0424 02:58:48.666348 1 periodic.go:263] Periodic gather workloads completed in 8.409s I0424 02:58:48.666356 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0424 02:58:48.666360 1 periodic.go:216] Running conditional gatherer I0424 02:58:48.672723 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0424 02:58:50.160962 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0424 02:58:51.258641 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0424 02:58:51.265227 1 conditional_gatherer.go:392] cluster version is '4.21.9' I0424 02:58:51.265255 1 tasks_processing.go:45] number of workers: 3 I0424 02:58:51.265262 1 tasks_processing.go:69] worker 2 listening for tasks. I0424 02:58:51.265267 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0424 02:58:51.265330 1 tasks_processing.go:69] worker 1 listening for tasks. I0424 02:58:51.265343 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0424 02:58:51.265344 1 tasks_processing.go:71] worker 1 working on remote_configuration task. I0424 02:58:51.265345 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0424 02:58:51.265352 1 tasks_processing.go:69] worker 0 listening for tasks. I0424 02:58:51.265353 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 963ns to process 1 records I0424 02:58:51.265356 1 tasks_processing.go:74] worker 0 stopped. I0424 02:58:51.265385 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0424 02:58:51.265392 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.1µs to process 1 records I0424 02:58:51.265398 1 tasks_processing.go:74] worker 1 stopped. I0424 02:58:51.265623 1 tasks_processing.go:74] worker 2 stopped. I0424 02:58:51.265660 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 264.365µs to process 0 records I0424 02:58:51.265720 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0424 02:58:51.265754 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0424 02:58:51.297019 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=dbe8da32e01e929235f1f50f6a0c0a948c2a84f8f515f54767b4a287b0b944b6 I0424 02:58:51.297148 1 diskrecorder.go:70] Writing 110 records to /var/lib/insights-operator/insights-2026-04-24-025851.tar.gz I0424 02:58:51.303695 1 diskrecorder.go:51] Wrote 110 records to disk in 6ms I0424 02:58:51.303721 1 periodic.go:285] Gathering cluster info every 2h0m0s I0424 02:58:51.303739 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0424 02:59:56.647012 1 diskrecorder.go:223] Found files to send: insights-2026-04-24-025851.tar.gz I0424 02:59:56.647041 1 insightsuploader.go:150] Checking archives to upload periodically every 15m38.733141312s I0424 02:59:56.647052 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0424 02:59:56.661618 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0424 02:59:56.921861 1 requests.go:87] Successfully reported id=2026-04-24T02:59:56Z x-rh-insights-request-id=7918eaf4a56940f9975ef5ad8487dfd2, wrote=48115 I0424 02:59:56.921912 1 insightsuploader.go:187] Uploaded report successfully in 274.854061ms I0424 02:59:56.921948 1 controller.go:129] Initializing last reported time to 2026-04-24T02:59:56Z I0424 02:59:56.921974 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0424 02:59:56.921983 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0424 02:59:56.921990 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0424 02:59:56.928266 1 controller.go:554] The operator is healthy I0424 03:00:26.662641 1 controller.go:554] The operator is healthy I0424 03:01:01.104892 1 insightsreport.go:137] Pulling report from smart-proxy I0424 03:01:01.104917 1 insightsreport.go:149] Retrieving report I0424 03:01:01.112002 1 requests.go:110] Retrieving report for cluster: ff3c42e6-52a2-4333-9783-b331ebf4570a I0424 03:01:01.112013 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/ff3c42e6-52a2-4333-9783-b331ebf4570a/reports I0424 03:01:01.115083 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/ff3c42e6-52a2-4333-9783-b331ebf4570a/reports I0424 03:01:01.335822 1 insightsreport.go:184] Report retrieved I0424 03:01:01.368834 1 insightsreport.go:239] Report retrieved correctly I0424 03:02:26.660821 1 controller.go:554] The operator is healthy I0424 03:03:26.638457 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0424 03:03:26.645213 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 03:03:26.645230 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0424 03:03:26.650774 1 secretconfigobserver.go:119] support secret does not exist I0424 03:04:26.661806 1 controller.go:554] The operator is healthy I0424 03:06:26.662555 1 controller.go:554] The operator is healthy I0424 03:08:26.634192 1 configmapobserver.go:84] configmaps "insights-config" not found I0424 03:08:26.651310 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0424 03:08:26.656485 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 03:08:26.656501 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0424 03:08:26.660584 1 controller.go:554] The operator is healthy I0424 03:08:26.660994 1 secretconfigobserver.go:119] support secret does not exist I0424 03:10:26.664585 1 controller.go:554] The operator is healthy I0424 03:12:26.663825 1 controller.go:554] The operator is healthy I0424 03:13:26.661799 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0424 03:13:26.667607 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 03:13:26.667626 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0424 03:13:26.672554 1 secretconfigobserver.go:119] support secret does not exist I0424 03:14:26.660794 1 controller.go:554] The operator is healthy I0424 03:15:40.656556 1 insightsuploader.go:146] Nothing to report since 2026-04-24T02:59:56Z I0424 03:16:26.661230 1 controller.go:554] The operator is healthy I0424 03:18:26.635847 1 configmapobserver.go:84] configmaps "insights-config" not found I0424 03:18:26.640541 1 configmapobserver.go:84] configmaps "insights-config" not found I0424 03:18:26.659516 1 controller.go:554] The operator is healthy I0424 03:18:26.673401 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0424 03:18:26.678691 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 03:18:26.678707 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0424 03:18:26.683216 1 secretconfigobserver.go:119] support secret does not exist I0424 03:20:26.660989 1 controller.go:554] The operator is healthy I0424 03:22:26.661805 1 controller.go:554] The operator is healthy I0424 03:23:26.684253 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0424 03:23:26.690209 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0424 03:23:26.690226 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0424 03:23:26.695117 1 secretconfigobserver.go:119] support secret does not exist I0424 03:24:26.661602 1 controller.go:554] The operator is healthy