I0428 11:04:47.497508 1 cmd.go:253] Using service-serving-cert provided certificates I0428 11:04:47.497925 1 observer_polling.go:159] Starting file observer I0428 11:04:48.114395 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0428 11:04:48.114608 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0428 11:04:48.115143 1 secure_serving.go:57] Forcing use of http/1.1 only W0428 11:04:48.115163 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0428 11:04:48.115168 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0428 11:04:48.115171 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0428 11:04:48.115174 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0428 11:04:48.115176 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0428 11:04:48.115178 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0428 11:04:48.115273 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0428 11:04:48.119497 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0428 11:04:48.119498 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"967c5520-5b9e-416f-b135-aac057a1928f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0428 11:04:48.121420 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0428 11:04:48.121438 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0428 11:04:48.121422 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0428 11:04:48.121457 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0428 11:04:48.121393 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0428 11:04:48.121487 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0428 11:04:48.121763 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0428 11:04:48.122139 1 secure_serving.go:213] Serving securely on [::]:8443 I0428 11:04:48.122161 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0428 11:04:48.129598 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0428 11:04:48.129622 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0428 11:04:48.129651 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0428 11:04:48.135456 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0428 11:04:48.135473 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0428 11:04:48.140036 1 secretconfigobserver.go:119] support secret does not exist I0428 11:04:48.144600 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0428 11:04:48.148654 1 secretconfigobserver.go:119] support secret does not exist I0428 11:04:48.150665 1 recorder.go:176] Pruning old reports every 6h36m41s, max age is 288h0m0s I0428 11:04:48.155406 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0428 11:04:48.155419 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0428 11:04:48.155407 1 periodic.go:216] Running clusterconfig gatherer I0428 11:04:48.155428 1 insightsreport.go:296] Starting report retriever I0428 11:04:48.155432 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0428 11:04:48.155421 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0428 11:04:48.155463 1 tasks_processing.go:45] number of workers: 64 I0428 11:04:48.155489 1 tasks_processing.go:69] worker 3 listening for tasks. I0428 11:04:48.155497 1 tasks_processing.go:69] worker 19 listening for tasks. I0428 11:04:48.155502 1 tasks_processing.go:69] worker 63 listening for tasks. I0428 11:04:48.155505 1 tasks_processing.go:69] worker 38 listening for tasks. I0428 11:04:48.155507 1 tasks_processing.go:71] worker 19 working on certificate_signing_requests task. I0428 11:04:48.155512 1 tasks_processing.go:69] worker 20 listening for tasks. I0428 11:04:48.155512 1 tasks_processing.go:69] worker 40 listening for tasks. I0428 11:04:48.155514 1 tasks_processing.go:69] worker 1 listening for tasks. I0428 11:04:48.155518 1 tasks_processing.go:69] worker 21 listening for tasks. I0428 11:04:48.155519 1 tasks_processing.go:69] worker 2 listening for tasks. I0428 11:04:48.155507 1 tasks_processing.go:69] worker 0 listening for tasks. I0428 11:04:48.155524 1 tasks_processing.go:69] worker 22 listening for tasks. I0428 11:04:48.155523 1 tasks_processing.go:69] worker 41 listening for tasks. I0428 11:04:48.155527 1 tasks_processing.go:69] worker 42 listening for tasks. I0428 11:04:48.155530 1 tasks_processing.go:69] worker 7 listening for tasks. I0428 11:04:48.155531 1 tasks_processing.go:69] worker 43 listening for tasks. I0428 11:04:48.155535 1 tasks_processing.go:69] worker 23 listening for tasks. I0428 11:04:48.155536 1 tasks_processing.go:69] worker 44 listening for tasks. I0428 11:04:48.155538 1 tasks_processing.go:69] worker 6 listening for tasks. I0428 11:04:48.155531 1 tasks_processing.go:69] worker 12 listening for tasks. I0428 11:04:48.155541 1 tasks_processing.go:69] worker 24 listening for tasks. I0428 11:04:48.155542 1 tasks_processing.go:69] worker 45 listening for tasks. I0428 11:04:48.155546 1 tasks_processing.go:69] worker 46 listening for tasks. I0428 11:04:48.155544 1 tasks_processing.go:69] worker 5 listening for tasks. I0428 11:04:48.155545 1 tasks_processing.go:69] worker 25 listening for tasks. I0428 11:04:48.155551 1 tasks_processing.go:69] worker 9 listening for tasks. I0428 11:04:48.155554 1 tasks_processing.go:69] worker 11 listening for tasks. I0428 11:04:48.155557 1 tasks_processing.go:69] worker 10 listening for tasks. I0428 11:04:48.155563 1 tasks_processing.go:69] worker 48 listening for tasks. I0428 11:04:48.155550 1 tasks_processing.go:69] worker 26 listening for tasks. I0428 11:04:48.155554 1 tasks_processing.go:69] worker 27 listening for tasks. I0428 11:04:48.155570 1 tasks_processing.go:69] worker 49 listening for tasks. I0428 11:04:48.155570 1 tasks_processing.go:69] worker 37 listening for tasks. I0428 11:04:48.155571 1 tasks_processing.go:69] worker 29 listening for tasks. I0428 11:04:48.155571 1 tasks_processing.go:69] worker 13 listening for tasks. I0428 11:04:48.155576 1 tasks_processing.go:69] worker 32 listening for tasks. I0428 11:04:48.155555 1 tasks_processing.go:69] worker 4 listening for tasks. I0428 11:04:48.155582 1 tasks_processing.go:69] worker 17 listening for tasks. I0428 11:04:48.155585 1 tasks_processing.go:69] worker 51 listening for tasks. I0428 11:04:48.155589 1 tasks_processing.go:69] worker 54 listening for tasks. I0428 11:04:48.155577 1 tasks_processing.go:69] worker 18 listening for tasks. I0428 11:04:48.155592 1 tasks_processing.go:69] worker 52 listening for tasks. I0428 11:04:48.155595 1 tasks_processing.go:69] worker 61 listening for tasks. I0428 11:04:48.155597 1 tasks_processing.go:69] worker 53 listening for tasks. I0428 11:04:48.155542 1 tasks_processing.go:69] worker 8 listening for tasks. I0428 11:04:48.155600 1 tasks_processing.go:69] worker 62 listening for tasks. I0428 11:04:48.155602 1 tasks_processing.go:71] worker 63 working on overlapping_namespace_uids task. I0428 11:04:48.155610 1 tasks_processing.go:71] worker 44 working on image_pruners task. I0428 11:04:48.155603 1 tasks_processing.go:69] worker 58 listening for tasks. I0428 11:04:48.155616 1 tasks_processing.go:71] worker 62 working on machine_sets task. I0428 11:04:48.155617 1 tasks_processing.go:71] worker 6 working on sap_pods task. I0428 11:04:48.155620 1 tasks_processing.go:71] worker 58 working on openstack_controlplanes task. I0428 11:04:48.155616 1 tasks_processing.go:71] worker 40 working on nodenetworkconfigurationpolicies task. I0428 11:04:48.155671 1 tasks_processing.go:71] worker 13 working on machine_healthchecks task. I0428 11:04:48.155559 1 tasks_processing.go:69] worker 28 listening for tasks. I0428 11:04:48.155508 1 tasks_processing.go:69] worker 39 listening for tasks. I0428 11:04:48.155928 1 tasks_processing.go:69] worker 55 listening for tasks. I0428 11:04:48.155565 1 tasks_processing.go:69] worker 14 listening for tasks. I0428 11:04:48.155611 1 tasks_processing.go:71] worker 25 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0428 11:04:48.156053 1 tasks_processing.go:71] worker 53 working on proxies task. I0428 11:04:48.156126 1 tasks_processing.go:71] worker 52 working on cluster_apiserver task. I0428 11:04:48.156183 1 tasks_processing.go:71] worker 39 working on pdbs task. I0428 11:04:48.156296 1 tasks_processing.go:71] worker 43 working on install_plans task. I0428 11:04:48.156354 1 tasks_processing.go:71] worker 51 working on lokistack task. I0428 11:04:48.156466 1 tasks_processing.go:71] worker 2 working on machines task. I0428 11:04:48.155938 1 tasks_processing.go:69] worker 56 listening for tasks. I0428 11:04:48.156659 1 tasks_processing.go:71] worker 56 working on aggregated_monitoring_cr_names task. I0428 11:04:48.156759 1 tasks_processing.go:71] worker 49 working on jaegers task. I0428 11:04:48.156308 1 tasks_processing.go:71] worker 9 working on openstack_dataplanedeployments task. I0428 11:04:48.155558 1 tasks_processing.go:69] worker 47 listening for tasks. I0428 11:04:48.156211 1 tasks_processing.go:71] worker 17 working on sap_datahubs task. I0428 11:04:48.156957 1 tasks_processing.go:71] worker 21 working on openstack_version task. I0428 11:04:48.156988 1 tasks_processing.go:71] worker 55 working on version task. I0428 11:04:48.156058 1 tasks_processing.go:71] worker 42 working on sap_config task. I0428 11:04:48.157192 1 tasks_processing.go:71] worker 1 working on clusterroles task. I0428 11:04:48.157389 1 tasks_processing.go:71] worker 37 working on config_maps task. I0428 11:04:48.157507 1 tasks_processing.go:71] worker 47 working on infrastructures task. I0428 11:04:48.157716 1 tasks_processing.go:71] worker 27 working on machine_configs task. I0428 11:04:48.157764 1 tasks_processing.go:71] worker 29 working on container_images task. I0428 11:04:48.157808 1 tasks_processing.go:71] worker 45 working on mutating_webhook_configurations task. I0428 11:04:48.156662 1 tasks_processing.go:71] worker 28 working on node_features task. I0428 11:04:48.157816 1 tasks_processing.go:71] worker 0 working on qemu_kubevirt_launcher_logs task. I0428 11:04:48.156383 1 tasks_processing.go:71] worker 48 working on dvo_metrics task. I0428 11:04:48.157718 1 tasks_processing.go:71] worker 14 working on machine_autoscalers task. I0428 11:04:48.155500 1 tasks_processing.go:71] worker 3 working on image task. I0428 11:04:48.155577 1 tasks_processing.go:69] worker 31 listening for tasks. I0428 11:04:48.155581 1 tasks_processing.go:69] worker 36 listening for tasks. I0428 11:04:48.155583 1 tasks_processing.go:69] worker 30 listening for tasks. I0428 11:04:48.155585 1 tasks_processing.go:69] worker 60 listening for tasks. I0428 11:04:48.155587 1 tasks_processing.go:69] worker 57 listening for tasks. I0428 11:04:48.155591 1 tasks_processing.go:69] worker 35 listening for tasks. I0428 11:04:48.155593 1 tasks_processing.go:69] worker 59 listening for tasks. I0428 11:04:48.155606 1 tasks_processing.go:71] worker 38 working on metrics task. I0428 11:04:48.155564 1 tasks_processing.go:69] worker 33 listening for tasks. I0428 11:04:48.155563 1 tasks_processing.go:69] worker 16 listening for tasks. I0428 11:04:48.155919 1 tasks_processing.go:71] worker 20 working on cost_management_metrics_configs task. I0428 11:04:48.155932 1 tasks_processing.go:71] worker 41 working on service_accounts task. I0428 11:04:48.155970 1 tasks_processing.go:71] worker 32 working on ingress_certificates task. I0428 11:04:48.155997 1 tasks_processing.go:71] worker 18 working on networks task. I0428 11:04:48.158679 1 tasks_processing.go:71] worker 54 working on tsdb_status task. I0428 11:04:48.155711 1 tasks_processing.go:71] worker 22 working on openshift_logging task. I0428 11:04:48.156018 1 tasks_processing.go:71] worker 4 working on oauths task. I0428 11:04:48.156094 1 tasks_processing.go:71] worker 61 working on storage_cluster task. I0428 11:04:48.156176 1 tasks_processing.go:71] worker 8 working on feature_gates task. I0428 11:04:48.156223 1 tasks_processing.go:71] worker 7 working on storage_classes task. I0428 11:04:48.156230 1 tasks_processing.go:71] worker 26 working on machine_config_pools task. I0428 11:04:48.159071 1 tasks_processing.go:71] worker 30 working on nodes task. I0428 11:04:48.156314 1 tasks_processing.go:71] worker 11 working on nodenetworkstates task. I0428 11:04:48.156319 1 tasks_processing.go:71] worker 10 working on validating_webhook_configurations task. I0428 11:04:48.159195 1 tasks_processing.go:71] worker 60 working on node_logs task. I0428 11:04:48.159208 1 tasks_processing.go:71] worker 57 working on schedulers task. I0428 11:04:48.156720 1 tasks_processing.go:71] worker 23 working on authentication task. I0428 11:04:48.159323 1 tasks_processing.go:71] worker 59 working on monitoring_persistent_volumes task. I0428 11:04:48.159341 1 tasks_processing.go:71] worker 33 working on container_runtime_configs task. I0428 11:04:48.155570 1 tasks_processing.go:69] worker 15 listening for tasks. I0428 11:04:48.159363 1 tasks_processing.go:71] worker 15 working on silenced_alerts task. I0428 11:04:48.157789 1 tasks_processing.go:71] worker 12 working on crds task. I0428 11:04:48.159377 1 tasks_processing.go:71] worker 16 working on ingress task. I0428 11:04:48.155577 1 tasks_processing.go:69] worker 50 listening for tasks. I0428 11:04:48.157803 1 tasks_processing.go:71] worker 24 working on openstack_dataplanenodesets task. I0428 11:04:48.157844 1 tasks_processing.go:71] worker 46 working on image_registries task. I0428 11:04:48.155565 1 tasks_processing.go:69] worker 34 listening for tasks. I0428 11:04:48.157979 1 tasks_processing.go:71] worker 5 working on olm_operators task. I0428 11:04:48.159061 1 tasks_processing.go:71] worker 31 working on operators task. I0428 11:04:48.159432 1 tasks_processing.go:71] worker 50 working on pod_network_connectivity_checks task. I0428 11:04:48.159067 1 tasks_processing.go:71] worker 36 working on support_secret task. I0428 11:04:48.159317 1 tasks_processing.go:71] worker 35 working on active_alerts task. I0428 11:04:48.159614 1 tasks_processing.go:71] worker 34 working on operators_pods_and_events task. I0428 11:04:48.161323 1 tasks_processing.go:71] worker 51 working on openshift_machine_api_events task. I0428 11:04:48.161355 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 4.895438ms to process 0 records I0428 11:04:48.161641 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0428 11:04:48.161661 1 controller.go:368] The initial operator extension status is healthy I0428 11:04:48.161673 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0428 11:04:48.161681 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0428 11:04:48.161685 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0428 11:04:48.161704 1 controller.go:531] The operator is still being initialized I0428 11:04:48.161711 1 controller.go:554] The operator is healthy I0428 11:04:48.161840 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0428 11:04:48.161857 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0428 11:04:48.161902 1 operator.go:328] started I0428 11:04:48.161924 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0428 11:04:48.163434 1 tasks_processing.go:71] worker 17 working on ceph_cluster task. I0428 11:04:48.163450 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 6.549556ms to process 0 records I0428 11:04:48.163464 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 6.598861ms to process 0 records I0428 11:04:48.163469 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 6.475478ms to process 0 records I0428 11:04:48.163512 1 tasks_processing.go:74] worker 9 stopped. I0428 11:04:48.163512 1 tasks_processing.go:74] worker 21 stopped. I0428 11:04:48.163754 1 tasks_processing.go:74] worker 44 stopped. I0428 11:04:48.163972 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=872ffaa20938187cf8244dabd8bd64644fcd8447b050bb303464dafb3799dcce I0428 11:04:48.163985 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 8.132309ms to process 1 records E0428 11:04:48.168160 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host I0428 11:04:48.168176 1 tasks_processing.go:74] worker 54 stopped. E0428 11:04:48.168183 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host E0428 11:04:48.168178 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host I0428 11:04:48.168194 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 9.482113ms to process 0 records E0428 11:04:48.168205 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host I0428 11:04:48.168219 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 9.958958ms to process 0 records I0428 11:04:48.168236 1 tasks_processing.go:74] worker 38 stopped. E0428 11:04:48.168859 1 gather_active_alerts.go:64] Unable to retrieve most recent alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host I0428 11:04:48.168877 1 tasks_processing.go:74] worker 35 stopped. E0428 11:04:48.168883 1 gather.go:140] gatherer "clusterconfig" function "active_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host I0428 11:04:48.168891 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 9.210706ms to process 0 records E0428 11:04:48.168924 1 gather_silenced_alerts.go:52] Unable to retrieve silenced alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host I0428 11:04:48.168940 1 tasks_processing.go:74] worker 15 stopped. E0428 11:04:48.168950 1 gather.go:140] gatherer "clusterconfig" function "silenced_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host I0428 11:04:48.168965 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 9.568256ms to process 0 records I0428 11:04:48.175993 1 tasks_processing.go:74] worker 49 stopped. I0428 11:04:48.176004 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 19.219005ms to process 0 records I0428 11:04:48.179445 1 tasks_processing.go:74] worker 58 stopped. I0428 11:04:48.179459 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 23.813342ms to process 0 records I0428 11:04:48.179470 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 22.301293ms to process 0 records I0428 11:04:48.179476 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 23.826912ms to process 0 records I0428 11:04:48.179480 1 tasks_processing.go:74] worker 42 stopped. I0428 11:04:48.179482 1 tasks_processing.go:74] worker 40 stopped. I0428 11:04:48.179578 1 tasks_processing.go:74] worker 2 stopped. E0428 11:04:48.179588 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0428 11:04:48.179595 1 gather.go:177] gatherer "clusterconfig" function "machines" took 23.066703ms to process 0 records I0428 11:04:48.179613 1 tasks_processing.go:74] worker 6 stopped. I0428 11:04:48.179623 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 23.986612ms to process 0 records I0428 11:04:48.179632 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 20.837552ms to process 0 records I0428 11:04:48.179640 1 tasks_processing.go:74] worker 22 stopped. I0428 11:04:48.180059 1 tasks_processing.go:74] worker 14 stopped. I0428 11:04:48.180093 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 21.926937ms to process 0 records I0428 11:04:48.180473 1 tasks_processing.go:74] worker 18 stopped. I0428 11:04:48.180744 1 recorder.go:75] Recording config/network with fingerprint=2ca6d9363d0f31a5e9f26e6ea21da45ebed2e2a68d18b16a6d16bc07ff238e64 I0428 11:04:48.180773 1 gather.go:177] gatherer "clusterconfig" function "networks" took 21.717597ms to process 1 records I0428 11:04:48.181367 1 tasks_processing.go:74] worker 47 stopped. E0428 11:04:48.181906 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0428 11:04:48.182372 1 recorder.go:75] Recording config/infrastructure with fingerprint=07b3bc48b540b9a072f9235054f9f0021b774836ea5a4b84579da49e2f547a0b I0428 11:04:48.182388 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 22.886053ms to process 1 records I0428 11:04:48.182481 1 tasks_processing.go:74] worker 7 stopped. I0428 11:04:48.182867 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=6664abe16b43329495a72f21d1effb1dbeb7d53a0cab0d4dd01924842c3b2ae3 I0428 11:04:48.182888 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=03f12ce81103eec2afb435eadaa5f0062889be81a9b14f0c8d46ac67241fb768 I0428 11:04:48.182894 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 21.561743ms to process 2 records I0428 11:04:48.182904 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 24.954957ms to process 0 records E0428 11:04:48.182909 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0428 11:04:48.182916 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 24.934455ms to process 0 records I0428 11:04:48.182917 1 tasks_processing.go:74] worker 62 stopped. I0428 11:04:48.182929 1 tasks_processing.go:74] worker 13 stopped. I0428 11:04:48.182989 1 tasks_processing.go:74] worker 52 stopped. I0428 11:04:48.183021 1 recorder.go:75] Recording config/apiserver with fingerprint=ccf335810d452fcc3097f9106038875a157a81b460f1185fa958bffdeba965fd I0428 11:04:48.183029 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 24.868332ms to process 1 records I0428 11:04:48.183036 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 22.163141ms to process 0 records I0428 11:04:48.183045 1 tasks_processing.go:74] worker 11 stopped. I0428 11:04:48.183071 1 recorder.go:75] Recording config/proxy with fingerprint=250fdcc07125c9268149c2750c7b2d5a0fb0e91c8f3206e21b1d43aefadf9947 I0428 11:04:48.183079 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 25.336019ms to process 1 records I0428 11:04:48.183094 1 tasks_processing.go:74] worker 53 stopped. I0428 11:04:48.183176 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=a2995ca86eab11975ca10633884f156cab74981c437335c80fe5d084cd677109 I0428 11:04:48.183180 1 tasks_processing.go:74] worker 39 stopped. I0428 11:04:48.183192 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=31d2c026553e9fad42e07833540ab7a76c6b2f82e1e912692bcb14271d6dcfeb I0428 11:04:48.183210 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=9902efa2a2630dd993ed132e8f990464e3eb1c3716aa887547964ca8cc59a2b5 I0428 11:04:48.183222 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=a9b46ab3a8e024a3983e2058e1867f1322e6161fbd73c29ebee4ab642a87e180 I0428 11:04:48.183237 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=ebd831b47af7a165e0eef750440162d213637d1f80630321b31335735a97f952 I0428 11:04:48.183251 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=a4214291c521f7faf6a55a4eac18b8558024f83c7d96454add25cdae7a9e6ed9 I0428 11:04:48.183257 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 25.56223ms to process 6 records I0428 11:04:48.183265 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 23.561502ms to process 0 records I0428 11:04:48.183269 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 22.482101ms to process 0 records I0428 11:04:48.183278 1 tasks_processing.go:74] worker 20 stopped. E0428 11:04:48.183280 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0428 11:04:48.183293 1 tasks_processing.go:74] worker 33 stopped. I0428 11:04:48.183297 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 22.459992ms to process 0 records I0428 11:04:48.183307 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 23.892759ms to process 0 records I0428 11:04:48.183312 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 22.988078ms to process 0 records I0428 11:04:48.183354 1 tasks_processing.go:74] worker 28 stopped. I0428 11:04:48.183285 1 tasks_processing.go:74] worker 50 stopped. I0428 11:04:48.183362 1 tasks_processing.go:74] worker 61 stopped. I0428 11:04:48.183376 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=194a4a8c90014babd247a9e5d2abb2145af7bd47ed8516e8e839afcea7e64ae8 I0428 11:04:48.183384 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 22.763402ms to process 1 records I0428 11:04:48.183426 1 tasks_processing.go:74] worker 57 stopped. I0428 11:04:48.183492 1 tasks_processing.go:74] worker 4 stopped. I0428 11:04:48.183585 1 recorder.go:75] Recording config/oauth with fingerprint=483421e650f6bce50c8523a060e9d083e074e2b242589617b0bffafdab578fac I0428 11:04:48.183605 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 23.525058ms to process 1 records I0428 11:04:48.185361 1 tasks_processing.go:74] worker 24 stopped. I0428 11:04:48.185378 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 25.861549ms to process 0 records I0428 11:04:48.185388 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 26.309013ms to process 0 records I0428 11:04:48.185395 1 tasks_processing.go:74] worker 26 stopped. I0428 11:04:48.185480 1 gather_logs.go:145] no pods in namespace were found I0428 11:04:48.185500 1 tasks_processing.go:74] worker 0 stopped. I0428 11:04:48.185509 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 27.422643ms to process 0 records I0428 11:04:48.185579 1 tasks_processing.go:74] worker 3 stopped. I0428 11:04:48.185686 1 recorder.go:75] Recording config/image with fingerprint=b041e4ae0c0bed824f0d0b3abeab9f5d4fa73c1ae16292de9a25c877b71f60aa I0428 11:04:48.185699 1 gather.go:177] gatherer "clusterconfig" function "image" took 27.392026ms to process 1 records I0428 11:04:48.187087 1 tasks_processing.go:74] worker 36 stopped. E0428 11:04:48.187120 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0428 11:04:48.187128 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 27.45109ms to process 0 records I0428 11:04:48.187397 1 tasks_processing.go:74] worker 27 stopped. E0428 11:04:48.187410 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0428 11:04:48.187429 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0428 11:04:48.187438 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 29.62561ms to process 1 records I0428 11:04:48.187576 1 tasks_processing.go:74] worker 45 stopped. I0428 11:04:48.187740 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=8aa7b31c1ea7fc548a9208ce554e6002e629784523aa32f1ee11dc6f0edadd4c I0428 11:04:48.187786 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=c81c6ec080188f87122112cd625604ee0e9218287357bfcc4e8034a44bebf975 I0428 11:04:48.187834 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=4e78395d13fc463737705ec371b7c844e7a7eb95efe249c4580be71326851604 I0428 11:04:48.187842 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 29.569077ms to process 3 records I0428 11:04:48.187933 1 tasks_processing.go:74] worker 23 stopped. I0428 11:04:48.188012 1 recorder.go:75] Recording config/authentication with fingerprint=5dfdc74ab0cd53ca89a88b339066f3ab6da36022391053d478121fd7c99c97a4 I0428 11:04:48.188025 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 28.345452ms to process 1 records I0428 11:04:48.188122 1 tasks_processing.go:74] worker 8 stopped. I0428 11:04:48.188136 1 recorder.go:75] Recording config/featuregate with fingerprint=b90fab7e312846789d6527cdcd5f2506cb7c27c135231b65347d0570148bc7c4 I0428 11:04:48.188145 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 28.772038ms to process 1 records I0428 11:04:48.188219 1 tasks_processing.go:74] worker 16 stopped. I0428 11:04:48.188253 1 recorder.go:75] Recording config/ingress with fingerprint=d1b0e90ea2485401d42cb1676985443456d036112cce7083fd4acef21414e428 I0428 11:04:48.188262 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 28.451537ms to process 1 records I0428 11:04:48.188285 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0428 11:04:48.188294 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 32.529325ms to process 1 records I0428 11:04:48.188346 1 tasks_processing.go:74] worker 63 stopped. I0428 11:04:48.188430 1 tasks_processing.go:74] worker 46 stopped. I0428 11:04:48.188943 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=5e2d0dbb0d29a5d217ffb3f2ef229875963450a22da7355545edd065d86542fd I0428 11:04:48.189004 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 28.703921ms to process 1 records I0428 11:04:48.190766 1 tasks_processing.go:74] worker 19 stopped. I0428 11:04:48.190780 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 35.245639ms to process 0 records W0428 11:04:48.190872 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0428 11:04:48.191941 1 tasks_processing.go:74] worker 60 stopped. I0428 11:04:48.191959 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 32.725335ms to process 0 records I0428 11:04:48.194748 1 tasks_processing.go:74] worker 17 stopped. I0428 11:04:48.194762 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 31.297927ms to process 0 records I0428 11:04:48.195010 1 tasks_processing.go:74] worker 59 stopped. I0428 11:04:48.195021 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 35.677157ms to process 0 records I0428 11:04:48.195190 1 tasks_processing.go:74] worker 51 stopped. I0428 11:04:48.195203 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 33.85376ms to process 0 records I0428 11:04:48.195247 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0428 11:04:48.195258 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0428 11:04:48.195263 1 controller.go:254] Source scaController *sca.Controller is not ready I0428 11:04:48.195266 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0428 11:04:48.195269 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0428 11:04:48.195286 1 controller.go:531] The operator is still being initialized I0428 11:04:48.195293 1 controller.go:554] The operator is healthy I0428 11:04:48.195698 1 tasks_processing.go:74] worker 5 stopped. I0428 11:04:48.195708 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 36.19415ms to process 0 records I0428 11:04:48.198477 1 tasks_processing.go:74] worker 55 stopped. I0428 11:04:48.198927 1 recorder.go:75] Recording config/version with fingerprint=856cd1e883cd2d909ee2350cbacc098b5b707ca11bf484338e7a6f4205cd86b2 I0428 11:04:48.198950 1 recorder.go:75] Recording config/id with fingerprint=39fb723bdd23b757a66e626279a0d40d362bec248bb6666638f2511cc9287291 I0428 11:04:48.198959 1 gather.go:177] gatherer "clusterconfig" function "version" took 41.472464ms to process 2 records I0428 11:04:48.199047 1 tasks_processing.go:74] worker 10 stopped. I0428 11:04:48.199093 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=35888bfd5f29c8ad9113f650d5a7b34d23080d4688e312e3b37d5861e8d45afd I0428 11:04:48.199197 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=e85d578d7ffd906349e68aa9d4ecb7dc238442bed869e2f518a746025368a47c I0428 11:04:48.199342 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=28077513cf444dec0c43f6389572b24e8fcb56d8b1fe0cf09deaff2ffab88e96 I0428 11:04:48.199389 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=f86ce4c09f39bccc363dd5f97c637ec43706687f36b6fbb05a8cf6bcd6a1915e I0428 11:04:48.199431 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=8fe0e4b63bccf1fd4953a0d0dd4908ca7ce7e8f99d30f15fe5c9126eefb5c8b2 I0428 11:04:48.199473 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=d4738ee691080b4c90b2e32465865e42c3014bffd673510511e6b9b8df1e38fe I0428 11:04:48.199516 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=13799fcbc319f1863afdf5f2de30c07a1f98e918c88ac1cfbc4a1403d85ba2da I0428 11:04:48.199562 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=709fc3c3eb80e6767cf2c4d9a740f8979fc29f286df50aa913e0ddc483348dad I0428 11:04:48.199607 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=71d899698aeec702e25c7a51c9a98efd65fbaa2a2ca58c59c3696eb7a91e2ebe I0428 11:04:48.199666 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=a8a60719348adf59a185aabb515698a622a864674f6d9f75d5b33e8b6cfb0062 I0428 11:04:48.199703 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=ab0229d156319c364155d133e9d55d1c7c0009ca8694832d1f971d3b3e93d379 I0428 11:04:48.199740 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=9839b8be9bb158e724a23cead5ae26b3944687a80c32773c4acd854bad3875fa I0428 11:04:48.199772 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=a758b6dea0a9265bf05f5890a6094d6b054bd322815c481004a4d46cde962280 I0428 11:04:48.199782 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 39.610806ms to process 13 records I0428 11:04:48.199876 1 tasks_processing.go:74] worker 30 stopped. I0428 11:04:48.200112 1 recorder.go:75] Recording config/node/ip-10-0-0-142.ec2.internal with fingerprint=f44e7c69cc5d11d02d1cc672fcaa36ca7da890a097165e14ff541fbdc2b4195d I0428 11:04:48.200170 1 recorder.go:75] Recording config/node/ip-10-0-1-214.ec2.internal with fingerprint=0074241092ff26e18530afcf61426ede4eec00c5b2d54fd6a222f97f470b2e62 I0428 11:04:48.200221 1 recorder.go:75] Recording config/node/ip-10-0-2-127.ec2.internal with fingerprint=b67dbaa342e4c7f174a19d411066c0661fd63b6c0bbcfd79b122b13051703fc2 I0428 11:04:48.200228 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 40.21304ms to process 3 records I0428 11:04:48.204597 1 tasks_processing.go:74] worker 1 stopped. I0428 11:04:48.204755 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=14266ed40589bcd4c7d8bc2697099e7e5773de5d29e6db03b4a4af9fdad2d0f1 I0428 11:04:48.204833 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=2d7edbf831409ce409daa7518b3c41c1e540ce842a835cac742d73b1de032192 I0428 11:04:48.204841 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 47.333647ms to process 2 records I0428 11:04:48.204969 1 tasks_processing.go:74] worker 12 stopped. I0428 11:04:48.205341 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=339880c13a7ea117599cd22dc2cd74e93526e3995d0a945840ac1d1cfa0b6c1a I0428 11:04:48.205580 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=436ac2d1fe19f9311af4cbf74afb089595d25e3b828a0b79d7f72367e6a67b92 I0428 11:04:48.205589 1 gather.go:177] gatherer "clusterconfig" function "crds" took 45.22957ms to process 2 records I0428 11:04:48.206837 1 tasks_processing.go:74] worker 34 stopped. I0428 11:04:48.206851 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 47.154351ms to process 0 records I0428 11:04:48.208523 1 tasks_processing.go:74] worker 56 stopped. I0428 11:04:48.208542 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 51.789249ms to process 0 records I0428 11:04:48.213864 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0428 11:04:48.216083 1 tasks_processing.go:74] worker 29 stopped. I0428 11:04:48.217288 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-66dbbb6468-5fr79 with fingerprint=069c389947979eecdff5d715b5bcf1073987349628b35b8d364e1c1b90fd409c I0428 11:04:48.217358 1 recorder.go:75] Recording config/running_containers with fingerprint=f52264f152a2e83dc01cfdeb7c1b97fdb61706ec37b67705fb43cd40bc7ca4f5 I0428 11:04:48.217368 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 58.265959ms to process 2 records I0428 11:04:48.218656 1 tasks_processing.go:74] worker 25 stopped. I0428 11:04:48.218669 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 62.607946ms to process 0 records I0428 11:04:48.221539 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0428 11:04:48.221553 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0428 11:04:48.221569 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0428 11:04:48.221684 1 tasks_processing.go:74] worker 37 stopped. E0428 11:04:48.221699 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0428 11:04:48.221705 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0428 11:04:48.221709 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0428 11:04:48.221717 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0428 11:04:48.221771 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0428 11:04:48.221789 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0428 11:04:48.221797 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0428 11:04:48.221808 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0428 11:04:48.221845 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0428 11:04:48.221851 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0428 11:04:48.221857 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 64.244421ms to process 7 records I0428 11:04:48.227049 1 tasks_processing.go:74] worker 32 stopped. E0428 11:04:48.227064 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0428 11:04:48.227071 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2pv452ledagn57tf47lfif0ho4ffo09v-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2pv452ledagn57tf47lfif0ho4ffo09v-primary-cert-bundle-secret" not found I0428 11:04:48.227134 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=9683231f3c6c8c3ae6687ce21ea52fdaae4d8b919ec1df6a8651969ea1780a27 I0428 11:04:48.227162 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 68.7028ms to process 1 records I0428 11:04:48.229800 1 base_controller.go:82] Caches are synced for ConfigController I0428 11:04:48.229816 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0428 11:04:48.242650 1 configmapobserver.go:84] configmaps "insights-config" not found I0428 11:04:48.262801 1 base_controller.go:82] Caches are synced for LoggingSyncer I0428 11:04:48.262813 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0428 11:04:48.270695 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0428 11:04:48.274802 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0428 11:04:48.310775 1 cluster_transfer.go:102] no available accepted cluster transfer I0428 11:04:48.310791 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0428 11:04:48.363793 1 sca.go:266] etc-pki-entitlement secret successfully created I0428 11:04:48.363810 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated W0428 11:04:49.190690 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0428 11:04:50.190836 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0428 11:04:50.628667 1 tasks_processing.go:74] worker 31 stopped. I0428 11:04:50.628785 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=27be089564fb369492eeb4d0ecdcff880f3ba380d129cc520905ad994a6f68fe I0428 11:04:50.628831 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=b1d3a57201f472fba248e6e22732d03f4ebac77f3d7204455295e9c6f31f376a I0428 11:04:50.628863 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=868c0c40189dcf90960a76cd6abfa98e9c5a9fa9921532c922a1f12d4cfea483 I0428 11:04:50.628872 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0428 11:04:50.628893 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=ec9c32f5f54a278bfe098b0b789eeb7b9e72f6b1be4022e04e6f8b9da6ccf8f1 I0428 11:04:50.628911 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0428 11:04:50.628932 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=176378b8c7b110d7d8be05436bda0c364d4cdc179fa89a6aa669083c2b48774f I0428 11:04:50.628973 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=08a456c8a515836b2cc4e1bde449a419f18adc068c7b524bfd7027e84daa2184 I0428 11:04:50.628997 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=8f0843402d1fceb7245afe79c78b733feebd637d5d3ddf6ffcf69aeb020534a8 I0428 11:04:50.629024 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=dd6eed08658b71fef77ae9c73f2364f1ab7093277279eb2662fca8731c10f77a I0428 11:04:50.629317 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=7bbc077d9b15fb63ae738ca9e50d1a43ee84fcb50978dfd0ab26fa57dc07f982 I0428 11:04:50.629339 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=fbcd63a285b62e2d2a18b6283a6dc2b23d4c8931bc1b8033a66d3e25e0dc286e I0428 11:04:50.629347 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0428 11:04:50.629363 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=a7ada666e0b09f95ec463ef59ac1e067a9a083d6249861e20832af0912ee2c45 I0428 11:04:50.629373 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0428 11:04:50.629391 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=523134888e87b3e048e975d8584dd293df67d5309fdd42d4b1fc7d222c8233cd I0428 11:04:50.629400 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0428 11:04:50.629418 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=88df24f3a63790e22d682b62cadc29d95dd04b9486b38c199b664e44d9dc35ce I0428 11:04:50.629428 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0428 11:04:50.629449 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=42ad0299a3edd0d24dd2c9c936ccda36baaeab38eff835ad4ccc99a09a720b27 I0428 11:04:50.629548 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=8f6a5d2eb8437e785041f5ea906e96e9819bba2108fd10a1362743f3bc0a38f2 I0428 11:04:50.629557 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0428 11:04:50.629564 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0428 11:04:50.629595 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0428 11:04:50.629617 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=c7b9bb03365f84753f298ec3a3a4ddb32cfcee31f2d9d55ab380d1b376b05690 I0428 11:04:50.629641 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=6e5b4f1f6d27d05d3d79ad757e0bdf67d0c4465fe2db42b3775ec71418ddea84 I0428 11:04:50.629650 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0428 11:04:50.629665 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=eb3e9b169c93eed800811b830fdd166e1348124cc3e3f9d63453b51415c29f0c I0428 11:04:50.629674 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0428 11:04:50.629692 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=f7795646ca55736ff5c54a0cb1a57149190b8c950b436c8f981b0bcf81835c7a I0428 11:04:50.629701 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0428 11:04:50.629713 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=7c66adebe45d19de3204c22a6c1c93bcaa5f51d41198627c89e7d995704516e5 I0428 11:04:50.629725 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=3dd2ef229bd5fa6388d3054aa7da397f0d37e9b76760f3c4bfd2a82b1f2e9834 I0428 11:04:50.629739 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=2b84ada6f22f053bc94a1f353d625abc7ed86375be2ddaf9421ce97cc364f0f4 I0428 11:04:50.629759 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=0e4044e17ec71d8c767f5677a406356c667cc7c46bfb20917dcd5226e64d96de I0428 11:04:50.629768 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0428 11:04:50.629790 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=1356fc4df277a1648d438a5fa54b457b3aa97c394ee43a273c1efebe4cb3033e I0428 11:04:50.629806 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0428 11:04:50.629813 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0428 11:04:50.629820 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.469227731s to process 39 records W0428 11:04:51.190239 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0428 11:04:52.189682 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0428 11:04:53.190080 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0428 11:04:53.190118 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0428 11:04:53.190134 1 tasks_processing.go:74] worker 48 stopped. E0428 11:04:53.190143 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0428 11:04:53.190154 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0428 11:04:53.190164 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.032008617s to process 1 records I0428 11:05:00.994282 1 tasks_processing.go:74] worker 43 stopped. I0428 11:05:00.994320 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0428 11:05:00.994334 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.837922328s to process 1 records I0428 11:05:01.765156 1 tasks_processing.go:74] worker 41 stopped. I0428 11:05:01.765439 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=80e5b054934e37ddf232a6b52185c88ef3149965af63f70b0125fd661e8f707e I0428 11:05:01.765459 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.606877331s to process 1 records E0428 11:05:01.765523 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.61s with: function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"active_alerts\" failed with an error, function \"silenced_alerts\" failed with an error, function \"machines\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0428 11:05:01.766629 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "tsdb_status" failed with an error, function "metrics" failed with an error, function "active_alerts" failed with an error, function "silenced_alerts" failed with an error, function "machines" failed with an error, function "machine_healthchecks" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0428 11:05:01.766643 1 periodic.go:216] Running workloads gatherer I0428 11:05:01.766657 1 tasks_processing.go:45] number of workers: 2 I0428 11:05:01.766665 1 tasks_processing.go:69] worker 1 listening for tasks. I0428 11:05:01.766669 1 tasks_processing.go:71] worker 1 working on helmchart_info task. I0428 11:05:01.766677 1 tasks_processing.go:69] worker 0 listening for tasks. I0428 11:05:01.766762 1 tasks_processing.go:71] worker 0 working on workload_info task. I0428 11:05:01.773501 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-142.ec2.internal... I0428 11:05:01.773510 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-214.ec2.internal... I0428 11:05:01.773522 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-127.ec2.internal... I0428 11:05:01.796232 1 tasks_processing.go:74] worker 1 stopped. I0428 11:05:01.796247 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 29.551671ms to process 0 records I0428 11:05:07.670790 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 6s I0428 11:05:07.695154 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0428 11:05:07.702704 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (9ms) I0428 11:05:07.710946 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (8ms) I0428 11:05:07.718923 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (8ms) I0428 11:05:07.727047 1 gather_workloads_info.go:387] No image sha256:53ff8efdd8f563421ebc1859ab021a3bbc1306f82d4ff5d9c21bc60ec5f57898 (8ms) I0428 11:05:07.734825 1 gather_workloads_info.go:387] No image sha256:48f5559439bea030a6423652da448c25fff0c62ba293eb521843c1a8eb89ddb0 (8ms) I0428 11:05:07.742394 1 gather_workloads_info.go:398] Found image sha256:23204877f166455ce8f9de9b91a467eeda576f172d5f96124a68aa986c4775d6 (8ms) I0428 11:05:07.753299 1 gather_workloads_info.go:387] No image sha256:6c6d412ccf0c3d6a14645a79df6c4658c0f8725d7954cb83c3d2f0116499345a (11ms) I0428 11:05:07.761035 1 gather_workloads_info.go:387] No image sha256:3380fecebaeee139c1607fbcd9346cebc92806679568e2244a8905ef0f52aad2 (8ms) I0428 11:05:07.768892 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (8ms) I0428 11:05:07.776788 1 gather_workloads_info.go:387] No image sha256:15e1b5f763553e86d0230a751b0783e3b990438fd07cf96eed164bccb72ed564 (8ms) I0428 11:05:07.804407 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (28ms) I0428 11:05:07.903739 1 gather_workloads_info.go:387] No image sha256:9ef6dc311b880a6442b7a3a19006ddd955c7da36850bf000f5c9b68fd0862dd4 (99ms) I0428 11:05:08.009327 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (106ms) I0428 11:05:08.110820 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (101ms) I0428 11:05:08.203680 1 gather_workloads_info.go:387] No image sha256:0acfe677ed58031f8381cb8a4f2d8917ec4cfa95485e2a46529dffae54c75ace (93ms) I0428 11:05:08.304078 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (100ms) I0428 11:05:08.402829 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (99ms) I0428 11:05:08.503058 1 gather_workloads_info.go:387] No image sha256:5940cc34b322325618929b1e0eb3864dae73d54c8f5b474e218dfb28123a61c7 (100ms) I0428 11:05:08.603080 1 gather_workloads_info.go:387] No image sha256:d6fbe0075cbb12bfd287c973704fadf97154c7f73e370733d976a40835e9155a (100ms) I0428 11:05:08.704158 1 gather_workloads_info.go:387] No image sha256:5dd9945374379a0e87d0dfecb1d0c7a5abb9116d11f6ebdc86c289bc1fca5a8a (101ms) I0428 11:05:08.803545 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (99ms) I0428 11:05:08.903911 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (100ms) I0428 11:05:09.002628 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (99ms) I0428 11:05:09.102822 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (100ms) I0428 11:05:09.203593 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (101ms) I0428 11:05:09.305139 1 gather_workloads_info.go:387] No image sha256:bdeb1c455cffccdb6c90db238e9d371de06340b1d6105d47f66d87293f878ac1 (102ms) I0428 11:05:09.403162 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (98ms) I0428 11:05:09.504067 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (101ms) I0428 11:05:09.603384 1 gather_workloads_info.go:387] No image sha256:17aefdc2f4752c23dc8f0947bb01fc34635822eeb18f2d8a5eddf8211b84d450 (99ms) I0428 11:05:09.703847 1 gather_workloads_info.go:387] No image sha256:1eb6a6ee3c66ac15be6bc21bb6015c595c11c15b9dfd21bd9f5ce7f7b990ebaa (100ms) I0428 11:05:09.802413 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (99ms) I0428 11:05:09.906464 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (104ms) I0428 11:05:10.003880 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (97ms) I0428 11:05:10.106572 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (103ms) I0428 11:05:10.204310 1 gather_workloads_info.go:387] No image sha256:0b2e6536a0ca3afc5e64a218087a6287c74b77a359075a385f1949ca5bd51b22 (98ms) I0428 11:05:10.303940 1 gather_workloads_info.go:387] No image sha256:572ebf7c7fab1a518582f3ca46c866730423cce6e8a19f614dc15fd61bf0c65a (100ms) I0428 11:05:10.404145 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (100ms) I0428 11:05:10.504619 1 gather_workloads_info.go:387] No image sha256:15928a5b803fa00781b408dbc4c3bfa7c5287b18584ba31f817def8a4430c725 (100ms) I0428 11:05:10.603226 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (99ms) I0428 11:05:10.705433 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (102ms) I0428 11:05:10.805083 1 gather_workloads_info.go:398] Found image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (100ms) I0428 11:05:10.903727 1 gather_workloads_info.go:387] No image sha256:9d6d63dce784df5863e6a0ba4c11317a26c9d756d4e62d47759c97190a71cce5 (99ms) I0428 11:05:11.008660 1 gather_workloads_info.go:387] No image sha256:52db05084f2a2e7d7bf4040e4865df88ec4074efbbe59a78d861be2c5ab2aa40 (105ms) I0428 11:05:11.104618 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (96ms) I0428 11:05:11.204276 1 gather_workloads_info.go:387] No image sha256:5ff204630794311b0b37fa7b197b933ad85d76a481bf7cdb3bcbada08f0cdcbf (100ms) I0428 11:05:11.304556 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (100ms) I0428 11:05:11.405127 1 gather_workloads_info.go:387] No image sha256:a53c639f63de8981aebdbd9f3aa0c2f43d4ec66f7624459ed6d6ee163db5a9c1 (101ms) I0428 11:05:11.503518 1 gather_workloads_info.go:387] No image sha256:5d6a2b8d2508c7da71192661e7c0dcf1335f48b6336a20189abffbe79eeff962 (98ms) I0428 11:05:11.603686 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (100ms) I0428 11:05:11.603723 1 tasks_processing.go:74] worker 0 stopped. I0428 11:05:11.604010 1 recorder.go:75] Recording config/workload_info with fingerprint=5ce0106015734aa4415f692d76d5665a561b5877d8160cb065598d82edd177e3 I0428 11:05:11.604027 1 gather.go:177] gatherer "workloads" function "workload_info" took 9.836944507s to process 1 records I0428 11:05:11.604042 1 periodic.go:263] Periodic gather workloads completed in 9.837s I0428 11:05:11.604051 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0428 11:05:11.604056 1 periodic.go:216] Running conditional gatherer I0428 11:05:11.609814 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0428 11:05:11.909022 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer E0428 11:05:11.911593 1 conditional_gatherer.go:324] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp 172.30.37.143:9091: connect: connection refused I0428 11:05:11.911671 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0428 11:05:11.917514 1 conditional_gatherer.go:392] cluster version is '4.21.9' E0428 11:05:11.917530 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 11:05:11.917535 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 11:05:11.917539 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 11:05:11.917542 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 11:05:11.917545 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 11:05:11.917548 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 11:05:11.917550 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 11:05:11.917553 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 11:05:11.917557 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 11:05:11.917561 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0428 11:05:11.917574 1 tasks_processing.go:45] number of workers: 3 I0428 11:05:11.917586 1 tasks_processing.go:69] worker 2 listening for tasks. I0428 11:05:11.917595 1 tasks_processing.go:69] worker 0 listening for tasks. I0428 11:05:11.917605 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0428 11:05:11.917611 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0428 11:05:11.917613 1 tasks_processing.go:69] worker 1 listening for tasks. I0428 11:05:11.917624 1 tasks_processing.go:74] worker 1 stopped. I0428 11:05:11.917630 1 tasks_processing.go:71] worker 0 working on conditional_gatherer_rules task. I0428 11:05:11.917672 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0428 11:05:11.917689 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.25µs to process 1 records I0428 11:05:11.917759 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=9fafdcbf373f1f367f1d69551ea59114592cce1681189be0f3dde774e0530661 I0428 11:05:11.917768 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.697µs to process 1 records I0428 11:05:11.917771 1 tasks_processing.go:74] worker 0 stopped. I0428 11:05:11.917851 1 tasks_processing.go:74] worker 2 stopped. I0428 11:05:11.917862 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 237.04µs to process 0 records I0428 11:05:11.917883 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0428 11:05:11.917900 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0428 11:05:11.944771 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=abc73e9a8de04c19a9f7e21a544dc542f9b998eb6ca60b7a979284546bcd48bc I0428 11:05:11.944921 1 diskrecorder.go:70] Writing 103 records to /var/lib/insights-operator/insights-2026-04-28-110511.tar.gz I0428 11:05:11.951367 1 diskrecorder.go:51] Wrote 103 records to disk in 6ms I0428 11:05:11.951400 1 periodic.go:285] Gathering cluster info every 2h0m0s I0428 11:05:11.951416 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0428 11:06:18.155677 1 diskrecorder.go:223] Found files to send: insights-2026-04-28-110511.tar.gz I0428 11:06:18.155703 1 insightsuploader.go:150] Checking archives to upload periodically every 15m0.644055586s I0428 11:06:18.155712 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0428 11:06:18.165228 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0428 11:06:18.357651 1 requests.go:87] Successfully reported id=2026-04-28T11:06:18Z x-rh-insights-request-id=0d5d1793ca6a4462a9607122561f0722, wrote=46887 I0428 11:06:18.357729 1 insightsuploader.go:187] Uploaded report successfully in 202.009261ms I0428 11:06:18.357752 1 controller.go:129] Initializing last reported time to 2026-04-28T11:06:18Z I0428 11:06:18.357783 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0428 11:06:18.357791 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0428 11:06:18.357798 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0428 11:06:18.364292 1 controller.go:554] The operator is healthy I0428 11:06:48.168950 1 controller.go:554] The operator is healthy I0428 11:07:22.882084 1 insightsreport.go:137] Pulling report from smart-proxy I0428 11:07:22.882127 1 insightsreport.go:149] Retrieving report I0428 11:07:22.889286 1 requests.go:110] Retrieving report for cluster: 445a7397-6609-45ed-8c4a-24c6b2ffd61c I0428 11:07:22.889299 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/445a7397-6609-45ed-8c4a-24c6b2ffd61c/reports I0428 11:07:22.892448 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/445a7397-6609-45ed-8c4a-24c6b2ffd61c/reports I0428 11:07:23.094177 1 insightsreport.go:184] Report retrieved I0428 11:07:23.121784 1 insightsreport.go:239] Report retrieved correctly I0428 11:08:48.169357 1 controller.go:554] The operator is healthy I0428 11:09:48.149345 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0428 11:09:48.157254 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0428 11:09:48.157273 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0428 11:09:48.161515 1 secretconfigobserver.go:119] support secret does not exist I0428 11:10:48.169129 1 controller.go:554] The operator is healthy I0428 11:12:48.168867 1 controller.go:554] The operator is healthy I0428 11:14:48.146510 1 configmapobserver.go:84] configmaps "insights-config" not found I0428 11:14:48.161551 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0428 11:14:48.166415 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0428 11:14:48.166433 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0428 11:14:48.168192 1 controller.go:554] The operator is healthy I0428 11:14:48.170330 1 secretconfigobserver.go:119] support secret does not exist I0428 11:16:48.169692 1 controller.go:554] The operator is healthy I0428 11:18:48.169505 1 controller.go:554] The operator is healthy I0428 11:19:48.170644 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0428 11:19:48.176559 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0428 11:19:48.176578 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0428 11:19:48.180892 1 secretconfigobserver.go:119] support secret does not exist