I0331 03:58:13.940983 1 cmd.go:253] Using service-serving-cert provided certificates I0331 03:58:13.941404 1 observer_polling.go:159] Starting file observer I0331 03:58:14.590041 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0331 03:58:14.590227 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0331 03:58:14.590773 1 secure_serving.go:57] Forcing use of http/1.1 only W0331 03:58:14.590792 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0331 03:58:14.590797 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0331 03:58:14.590801 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0331 03:58:14.590804 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0331 03:58:14.590806 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0331 03:58:14.590809 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0331 03:58:14.590850 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0331 03:58:14.595254 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0331 03:58:14.595286 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"fa3592bb-2ea1-441c-b69b-cd0b54d8489d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0331 03:58:14.595370 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0331 03:58:14.595401 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0331 03:58:14.595400 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0331 03:58:14.595432 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0331 03:58:14.595431 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0331 03:58:14.595440 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0331 03:58:14.595437 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0331 03:58:14.595822 1 secure_serving.go:213] Serving securely on [::]:8443 I0331 03:58:14.595847 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0331 03:58:14.599577 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0331 03:58:14.599600 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0331 03:58:14.599691 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0331 03:58:14.606126 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0331 03:58:14.606142 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0331 03:58:14.609964 1 secretconfigobserver.go:119] support secret does not exist I0331 03:58:14.613631 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0331 03:58:14.617218 1 secretconfigobserver.go:119] support secret does not exist I0331 03:58:14.622587 1 recorder.go:161] Pruning old reports every 8h46m39s, max age is 288h0m0s I0331 03:58:14.627360 1 periodic.go:209] Running clusterconfig gatherer I0331 03:58:14.627503 1 tasks_processing.go:45] number of workers: 64 I0331 03:58:14.627565 1 tasks_processing.go:69] worker 63 listening for tasks. I0331 03:58:14.627579 1 tasks_processing.go:71] worker 63 working on ingress task. I0331 03:58:14.627676 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0331 03:58:14.627748 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0331 03:58:14.627794 1 tasks_processing.go:69] worker 0 listening for tasks. I0331 03:58:14.627803 1 tasks_processing.go:69] worker 1 listening for tasks. I0331 03:58:14.627794 1 tasks_processing.go:69] worker 32 listening for tasks. I0331 03:58:14.627813 1 tasks_processing.go:69] worker 33 listening for tasks. I0331 03:58:14.627822 1 tasks_processing.go:69] worker 34 listening for tasks. I0331 03:58:14.627819 1 tasks_processing.go:69] worker 2 listening for tasks. I0331 03:58:14.627830 1 tasks_processing.go:69] worker 35 listening for tasks. I0331 03:58:14.627836 1 tasks_processing.go:69] worker 3 listening for tasks. I0331 03:58:14.627839 1 tasks_processing.go:69] worker 36 listening for tasks. I0331 03:58:14.627845 1 tasks_processing.go:69] worker 4 listening for tasks. I0331 03:58:14.627847 1 tasks_processing.go:69] worker 37 listening for tasks. I0331 03:58:14.627853 1 tasks_processing.go:69] worker 5 listening for tasks. I0331 03:58:14.627860 1 tasks_processing.go:69] worker 38 listening for tasks. I0331 03:58:14.627861 1 tasks_processing.go:69] worker 6 listening for tasks. I0331 03:58:14.627866 1 tasks_processing.go:69] worker 39 listening for tasks. I0331 03:58:14.627870 1 tasks_processing.go:69] worker 7 listening for tasks. I0331 03:58:14.627873 1 tasks_processing.go:69] worker 40 listening for tasks. I0331 03:58:14.627877 1 tasks_processing.go:69] worker 8 listening for tasks. I0331 03:58:14.627881 1 tasks_processing.go:69] worker 41 listening for tasks. I0331 03:58:14.627885 1 tasks_processing.go:69] worker 9 listening for tasks. I0331 03:58:14.627889 1 tasks_processing.go:69] worker 42 listening for tasks. I0331 03:58:14.627896 1 tasks_processing.go:69] worker 43 listening for tasks. I0331 03:58:14.627897 1 tasks_processing.go:69] worker 10 listening for tasks. I0331 03:58:14.627904 1 tasks_processing.go:69] worker 11 listening for tasks. I0331 03:58:14.627908 1 tasks_processing.go:69] worker 44 listening for tasks. I0331 03:58:14.627912 1 tasks_processing.go:69] worker 12 listening for tasks. I0331 03:58:14.627919 1 tasks_processing.go:69] worker 45 listening for tasks. I0331 03:58:14.627921 1 tasks_processing.go:69] worker 13 listening for tasks. I0331 03:58:14.627926 1 tasks_processing.go:69] worker 14 listening for tasks. I0331 03:58:14.627931 1 tasks_processing.go:69] worker 46 listening for tasks. I0331 03:58:14.627935 1 tasks_processing.go:69] worker 15 listening for tasks. I0331 03:58:14.627939 1 tasks_processing.go:69] worker 47 listening for tasks. I0331 03:58:14.627943 1 tasks_processing.go:69] worker 16 listening for tasks. I0331 03:58:14.627947 1 tasks_processing.go:69] worker 48 listening for tasks. I0331 03:58:14.627950 1 tasks_processing.go:69] worker 17 listening for tasks. I0331 03:58:14.627954 1 tasks_processing.go:69] worker 49 listening for tasks. I0331 03:58:14.627961 1 tasks_processing.go:69] worker 18 listening for tasks. I0331 03:58:14.627962 1 tasks_processing.go:69] worker 50 listening for tasks. I0331 03:58:14.627969 1 tasks_processing.go:69] worker 19 listening for tasks. I0331 03:58:14.627971 1 tasks_processing.go:69] worker 51 listening for tasks. I0331 03:58:14.627977 1 tasks_processing.go:69] worker 20 listening for tasks. I0331 03:58:14.627982 1 tasks_processing.go:69] worker 52 listening for tasks. I0331 03:58:14.627985 1 tasks_processing.go:69] worker 27 listening for tasks. I0331 03:58:14.627983 1 tasks_processing.go:69] worker 26 listening for tasks. I0331 03:58:14.627990 1 tasks_processing.go:69] worker 53 listening for tasks. I0331 03:58:14.627995 1 tasks_processing.go:69] worker 28 listening for tasks. I0331 03:58:14.627998 1 tasks_processing.go:69] worker 54 listening for tasks. I0331 03:58:14.628003 1 tasks_processing.go:69] worker 29 listening for tasks. I0331 03:58:14.628007 1 tasks_processing.go:69] worker 55 listening for tasks. I0331 03:58:14.628008 1 tasks_processing.go:69] worker 23 listening for tasks. I0331 03:58:14.627993 1 tasks_processing.go:69] worker 21 listening for tasks. I0331 03:58:14.628022 1 tasks_processing.go:69] worker 61 listening for tasks. I0331 03:58:14.628019 1 tasks_processing.go:69] worker 31 listening for tasks. I0331 03:58:14.628032 1 tasks_processing.go:69] worker 25 listening for tasks. I0331 03:58:14.628032 1 tasks_processing.go:69] worker 30 listening for tasks. I0331 03:58:14.628033 1 tasks_processing.go:69] worker 62 listening for tasks. I0331 03:58:14.628042 1 tasks_processing.go:69] worker 57 listening for tasks. I0331 03:58:14.628046 1 tasks_processing.go:71] worker 3 working on openstack_dataplanedeployments task. I0331 03:58:14.628050 1 tasks_processing.go:71] worker 2 working on validating_webhook_configurations task. I0331 03:58:14.628015 1 tasks_processing.go:69] worker 56 listening for tasks. I0331 03:58:14.628055 1 tasks_processing.go:71] worker 32 working on machine_sets task. I0331 03:58:14.628054 1 tasks_processing.go:69] worker 58 listening for tasks. I0331 03:58:14.628064 1 tasks_processing.go:71] worker 5 working on openstack_version task. I0331 03:58:14.628064 1 tasks_processing.go:71] worker 6 working on olm_operators task. I0331 03:58:14.628070 1 tasks_processing.go:71] worker 38 working on clusterroles task. I0331 03:58:14.628068 1 tasks_processing.go:71] worker 33 working on sap_config task. I0331 03:58:14.628070 1 tasks_processing.go:71] worker 37 working on openstack_controlplanes task. I0331 03:58:14.628086 1 tasks_processing.go:71] worker 39 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0331 03:58:14.628043 1 tasks_processing.go:71] worker 0 working on schedulers task. I0331 03:58:14.628147 1 tasks_processing.go:71] worker 18 working on machines task. I0331 03:58:14.628158 1 tasks_processing.go:71] worker 29 working on tsdb_status task. I0331 03:58:14.628159 1 tasks_processing.go:71] worker 7 working on nodes task. I0331 03:58:14.628156 1 tasks_processing.go:71] worker 4 working on cost_management_metrics_configs task. I0331 03:58:14.628147 1 tasks_processing.go:71] worker 12 working on image_pruners task. I0331 03:58:14.628229 1 tasks_processing.go:71] worker 40 working on ceph_cluster task. I0331 03:58:14.628037 1 tasks_processing.go:71] worker 34 working on openstack_dataplanenodesets task. I0331 03:58:14.628019 1 tasks_processing.go:69] worker 24 listening for tasks. I0331 03:58:14.628277 1 tasks_processing.go:71] worker 24 working on dvo_metrics task. I0331 03:58:14.628304 1 tasks_processing.go:71] worker 52 working on storage_classes task. I0331 03:58:14.628348 1 tasks_processing.go:71] worker 50 working on qemu_kubevirt_launcher_logs task. I0331 03:58:14.628364 1 tasks_processing.go:71] worker 23 working on pdbs task. I0331 03:58:14.628044 1 tasks_processing.go:71] worker 35 working on operators task. I0331 03:58:14.628041 1 tasks_processing.go:71] worker 36 working on authentication task. I0331 03:58:14.628322 1 tasks_processing.go:71] worker 53 working on openshift_machine_api_events task. I0331 03:58:14.628654 1 tasks_processing.go:71] worker 51 working on sap_datahubs task. I0331 03:58:14.628665 1 tasks_processing.go:71] worker 8 working on overlapping_namespace_uids task. I0331 03:58:14.628026 1 tasks_processing.go:69] worker 60 listening for tasks. I0331 03:58:14.628081 1 tasks_processing.go:71] worker 58 working on storage_cluster task. I0331 03:58:14.628692 1 tasks_processing.go:71] worker 60 working on support_secret task. I0331 03:58:14.628867 1 tasks_processing.go:71] worker 21 working on sap_pods task. I0331 03:58:14.628901 1 tasks_processing.go:71] worker 20 working on pod_network_connectivity_checks task. I0331 03:58:14.628916 1 tasks_processing.go:71] worker 41 working on oauths task. I0331 03:58:14.628055 1 tasks_processing.go:69] worker 59 listening for tasks. I0331 03:58:14.629065 1 tasks_processing.go:71] worker 43 working on ingress_certificates task. I0331 03:58:14.629071 1 tasks_processing.go:71] worker 59 working on node_logs task. I0331 03:58:14.629159 1 tasks_processing.go:71] worker 57 working on machine_healthchecks task. I0331 03:58:14.629178 1 tasks_processing.go:71] worker 28 working on lokistack task. I0331 03:58:14.629321 1 tasks_processing.go:71] worker 10 working on container_images task. I0331 03:58:14.629354 1 tasks_processing.go:71] worker 49 working on metrics task. I0331 03:58:14.628049 1 tasks_processing.go:71] worker 1 working on infrastructures task. I0331 03:58:14.628028 1 tasks_processing.go:69] worker 22 listening for tasks. I0331 03:58:14.629325 1 tasks_processing.go:71] worker 11 working on certificate_signing_requests task. I0331 03:58:14.629470 1 tasks_processing.go:71] worker 22 working on service_accounts task. I0331 03:58:14.628327 1 tasks_processing.go:71] worker 27 working on cluster_apiserver task. I0331 03:58:14.628338 1 tasks_processing.go:71] worker 26 working on nodenetworkstates task. I0331 03:58:14.628344 1 tasks_processing.go:71] worker 19 working on proxies task. I0331 03:58:14.628353 1 tasks_processing.go:71] worker 25 working on install_plans task. I0331 03:58:14.628358 1 tasks_processing.go:71] worker 55 working on version task. I0331 03:58:14.629016 1 tasks_processing.go:71] worker 31 working on config_maps task. I0331 03:58:14.629025 1 tasks_processing.go:71] worker 30 working on crds task. I0331 03:58:14.629030 1 tasks_processing.go:71] worker 62 working on machine_config_pools task. I0331 03:58:14.629034 1 tasks_processing.go:71] worker 61 working on monitoring_persistent_volumes task. I0331 03:58:14.629053 1 tasks_processing.go:71] worker 9 working on openshift_logging task. I0331 03:58:14.629037 1 tasks_processing.go:71] worker 56 working on mutating_webhook_configurations task. I0331 03:58:14.629057 1 tasks_processing.go:71] worker 42 working on image_registries task. I0331 03:58:14.629337 1 tasks_processing.go:71] worker 47 working on container_runtime_configs task. I0331 03:58:14.629336 1 tasks_processing.go:71] worker 54 working on feature_gates task. I0331 03:58:14.629344 1 tasks_processing.go:71] worker 45 working on operators_pods_and_events task. I0331 03:58:14.629344 1 tasks_processing.go:71] worker 44 working on nodenetworkconfigurationpolicies task. I0331 03:58:14.629349 1 tasks_processing.go:71] worker 16 working on machine_autoscalers task. I0331 03:58:14.629349 1 tasks_processing.go:71] worker 13 working on silenced_alerts task. I0331 03:58:14.629351 1 tasks_processing.go:71] worker 17 working on aggregated_monitoring_cr_names task. I0331 03:58:14.629354 1 tasks_processing.go:71] worker 14 working on networks task. I0331 03:58:14.629354 1 tasks_processing.go:71] worker 48 working on machine_configs task. I0331 03:58:14.629358 1 tasks_processing.go:71] worker 46 working on image task. I0331 03:58:14.629362 1 tasks_processing.go:71] worker 15 working on jaegers task. I0331 03:58:14.629367 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0331 03:58:14.630884 1 insightsreport.go:296] Starting report retriever I0331 03:58:14.630891 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0331 03:58:14.632951 1 tasks_processing.go:71] worker 32 working on active_alerts task. I0331 03:58:14.632959 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 4.879293ms to process 0 records I0331 03:58:14.632971 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 4.879531ms to process 0 records I0331 03:58:14.632979 1 tasks_processing.go:74] worker 5 stopped. I0331 03:58:14.636962 1 tasks_processing.go:74] worker 52 stopped. I0331 03:58:14.637130 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=6103bdad8921073c3b2d04995e6167f5696a74ad01b56238f8a86d48196f1734 I0331 03:58:14.637158 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=c60b9bbe13aaf6aef42ca3ae4eded96702e211f3300f9a382aef2f464d377fd9 I0331 03:58:14.637169 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 8.603411ms to process 2 records I0331 03:58:14.637607 1 tasks_processing.go:74] worker 37 stopped. I0331 03:58:14.637625 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 9.512737ms to process 0 records I0331 03:58:14.637643 1 tasks_processing.go:74] worker 3 stopped. I0331 03:58:14.637650 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 9.588742ms to process 0 records I0331 03:58:14.637792 1 tasks_processing.go:74] worker 20 stopped. E0331 03:58:14.637806 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0331 03:58:14.637817 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 8.838745ms to process 0 records I0331 03:58:14.637825 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 9.548492ms to process 0 records I0331 03:58:14.637833 1 tasks_processing.go:74] worker 34 stopped. I0331 03:58:14.637909 1 tasks_processing.go:74] worker 18 stopped. E0331 03:58:14.637929 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0331 03:58:14.637940 1 gather.go:177] gatherer "clusterconfig" function "machines" took 9.745516ms to process 0 records I0331 03:58:14.638137 1 tasks_processing.go:74] worker 63 stopped. I0331 03:58:14.638270 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0331 03:58:14.638316 1 controller.go:326] The initial operator extension status is healthy I0331 03:58:14.638349 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0331 03:58:14.638361 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0331 03:58:14.638365 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0331 03:58:14.638450 1 controller.go:489] The operator is still being initialized I0331 03:58:14.638463 1 controller.go:512] The operator is healthy I0331 03:58:14.638478 1 recorder.go:75] Recording config/ingress with fingerprint=f5841e26e5c57e0aaa80ea854136a89a13be8e3c67030df4b8811a5b64a8b88e I0331 03:58:14.638493 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 10.549315ms to process 1 records I0331 03:58:14.638588 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0331 03:58:14.638607 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0331 03:58:14.638697 1 operator.go:288] started I0331 03:58:14.638718 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer E0331 03:58:14.643403 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host I0331 03:58:14.643424 1 tasks_processing.go:74] worker 29 stopped. E0331 03:58:14.643437 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host E0331 03:58:14.643438 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host I0331 03:58:14.643451 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 15.254883ms to process 0 records E0331 03:58:14.643466 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host I0331 03:58:14.643482 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 14.046087ms to process 0 records I0331 03:58:14.643493 1 tasks_processing.go:74] worker 49 stopped. I0331 03:58:14.656048 1 tasks_processing.go:74] worker 0 stopped. I0331 03:58:14.656191 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=5909d64e83f06389ee6d51e0c9de16006968402a9b130e22f8dab93e35a57424 I0331 03:58:14.656249 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 27.906598ms to process 1 records I0331 03:58:14.656401 1 tasks_processing.go:74] worker 2 stopped. I0331 03:58:14.656709 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=10e920df45d131fb70e76f61e0a1710aa9dfd67e81a61961c36625d5fbfc0f38 I0331 03:58:14.656809 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=699c317c0d4d41ddffa68b4b23b71686d16ad1074396f9ca8c940ef19655c7dc I0331 03:58:14.657061 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=a3ee2789d508368091db0d82ba734cf6d29079265778065b13bbfe840282b68b I0331 03:58:14.657434 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=72753ddeaff7c095963dd5e00ac457f2e859c94af200a1a6daeb4e7535e96b59 I0331 03:58:14.657459 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=0e11d7a6529f9250c20b69e44a3e8ad0416925dfb8e22272b8fe4c68cc4ece99 I0331 03:58:14.657488 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=52ef18f2c552ce9b07c64c188dbb6bf8658dd1821b339ab88e36b57ec4229b72 I0331 03:58:14.657512 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=2e9642c42081d935c6f88bd8b5efd53250a49b319c197831cd3de1b9de641ff5 I0331 03:58:14.657534 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=7697af7c2bf788611f38d53365c1cff641648aab103bc11ea9e49d56517a196e I0331 03:58:14.657556 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=fe6b94a2a893ace402dc0e47aaec5bcffe3e85ab96ae5076e08ad37d01ac0e96 I0331 03:58:14.657591 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=4cd2dc89e9563aa6ccc33d0e1bc011983c812875501748a9697b31c77bf10f46 I0331 03:58:14.657614 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=09a40cc06b5558c06c22280f7dc61371e58d55560e945c356a8b9e792354ba00 I0331 03:58:14.657640 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=02670ef06a6d3c2646d8a2d22e5fbb119be41d989a8bf781ab862b8825b836e4 I0331 03:58:14.657662 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=7169f9720197d6251f31fbc719ba8fc8157355c70ebb299fff72d3c030e65570 I0331 03:58:14.657670 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 28.314484ms to process 13 records E0331 03:58:14.663441 1 gather_active_alerts.go:64] Unable to retrieve most recent alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host I0331 03:58:14.663455 1 tasks_processing.go:74] worker 32 stopped. E0331 03:58:14.663461 1 gather.go:140] gatherer "clusterconfig" function "active_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host I0331 03:58:14.663469 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 30.491451ms to process 0 records E0331 03:58:14.663507 1 gather_silenced_alerts.go:52] Unable to retrieve silenced alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host I0331 03:58:14.663526 1 tasks_processing.go:74] worker 13 stopped. E0331 03:58:14.663533 1 gather.go:140] gatherer "clusterconfig" function "silenced_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host I0331 03:58:14.663541 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 32.879616ms to process 0 records I0331 03:58:14.696467 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0331 03:58:14.696476 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0331 03:58:14.696491 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0331 03:58:14.700643 1 base_controller.go:82] Caches are synced for ConfigController I0331 03:58:14.700654 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0331 03:58:14.708676 1 tasks_processing.go:74] worker 21 stopped. I0331 03:58:14.708689 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 79.792131ms to process 0 records I0331 03:58:14.708723 1 tasks_processing.go:74] worker 61 stopped. I0331 03:58:14.708739 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 78.494871ms to process 0 records I0331 03:58:14.708856 1 tasks_processing.go:74] worker 46 stopped. I0331 03:58:14.708926 1 recorder.go:75] Recording config/image with fingerprint=5c50fc3ae48b5a0ab71ad32c856e3dcb1703af3c7263e3b1bc84d72d1192cd48 I0331 03:58:14.708936 1 gather.go:177] gatherer "clusterconfig" function "image" took 78.028206ms to process 1 records I0331 03:58:14.711563 1 tasks_processing.go:74] worker 33 stopped. I0331 03:58:14.711573 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 83.476311ms to process 0 records I0331 03:58:14.711595 1 tasks_processing.go:74] worker 51 stopped. I0331 03:58:14.711607 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 82.926781ms to process 0 records I0331 03:58:14.711683 1 tasks_processing.go:74] worker 58 stopped. I0331 03:58:14.711695 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 82.996895ms to process 0 records I0331 03:58:14.711708 1 tasks_processing.go:74] worker 47 stopped. I0331 03:58:14.711712 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 81.260081ms to process 0 records I0331 03:58:14.711846 1 tasks_processing.go:74] worker 41 stopped. I0331 03:58:14.712035 1 recorder.go:75] Recording config/oauth with fingerprint=ae3da669768ab1a08488766fd2c8b4399738f62fa36411ee80cbcd6b0dfa3295 I0331 03:58:14.712049 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 82.915286ms to process 1 records I0331 03:58:14.712079 1 tasks_processing.go:74] worker 15 stopped. I0331 03:58:14.712089 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 81.22843ms to process 0 records I0331 03:58:14.712387 1 tasks_processing.go:74] worker 26 stopped. I0331 03:58:14.712395 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 82.412414ms to process 0 records I0331 03:58:14.712478 1 tasks_processing.go:74] worker 4 stopped. I0331 03:58:14.712491 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 84.300545ms to process 0 records I0331 03:58:14.712615 1 tasks_processing.go:74] worker 44 stopped. I0331 03:58:14.712625 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 82.049057ms to process 0 records I0331 03:58:14.712679 1 tasks_processing.go:74] worker 14 stopped. I0331 03:58:14.712806 1 recorder.go:75] Recording config/network with fingerprint=ecbaed9eead916b12bd888fae1904b13b87d356aec102f6407496eeedb86ae09 I0331 03:58:14.712818 1 gather.go:177] gatherer "clusterconfig" function "networks" took 81.981326ms to process 1 records I0331 03:58:14.712921 1 tasks_processing.go:74] worker 54 stopped. I0331 03:58:14.713009 1 recorder.go:75] Recording config/featuregate with fingerprint=43f0e5cdaa1d84d33553aee223023562bbe267c34699df109eb786947e844a2c I0331 03:58:14.713019 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 82.386374ms to process 1 records I0331 03:58:14.713354 1 gather_logs.go:145] no pods in namespace were found I0331 03:58:14.713367 1 tasks_processing.go:74] worker 50 stopped. I0331 03:58:14.713390 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 84.997961ms to process 0 records I0331 03:58:14.713686 1 tasks_processing.go:74] worker 59 stopped. I0331 03:58:14.713700 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 84.601137ms to process 0 records I0331 03:58:14.714126 1 tasks_processing.go:74] worker 62 stopped. I0331 03:58:14.714144 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 84.021948ms to process 0 records I0331 03:58:14.714180 1 tasks_processing.go:74] worker 27 stopped. I0331 03:58:14.714292 1 recorder.go:75] Recording config/apiserver with fingerprint=a72d1f62928c6025f57b56641acaa24c86f23fa2b73c0907606d94507e04c493 I0331 03:58:14.714306 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 84.549871ms to process 1 records I0331 03:58:14.714390 1 tasks_processing.go:74] worker 1 stopped. I0331 03:58:14.714932 1 configmapobserver.go:84] configmaps "insights-config" not found I0331 03:58:14.715754 1 recorder.go:75] Recording config/infrastructure with fingerprint=e1024f8d2dfff0c1beaf12ac88614cf3165e6154e8599ddf2af8120ac1d05151 I0331 03:58:14.715831 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 84.819112ms to process 1 records E0331 03:58:14.715871 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0331 03:58:14.715890 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 85.633808ms to process 0 records I0331 03:58:14.715906 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 83.846946ms to process 0 records I0331 03:58:14.715915 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 85.266998ms to process 0 records I0331 03:58:14.715922 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 84.220032ms to process 0 records E0331 03:58:14.715928 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0331 03:58:14.715944 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 85.303107ms to process 0 records I0331 03:58:14.716013 1 tasks_processing.go:74] worker 19 stopped. I0331 03:58:14.716025 1 tasks_processing.go:74] worker 28 stopped. I0331 03:58:14.716041 1 tasks_processing.go:74] worker 60 stopped. I0331 03:58:14.716047 1 tasks_processing.go:74] worker 16 stopped. I0331 03:58:14.716063 1 tasks_processing.go:74] worker 9 stopped. I0331 03:58:14.716069 1 recorder.go:75] Recording config/proxy with fingerprint=1b90e0c2a54c8bec6fbffbce496eee2699328d643f1989830ffc35026bece717 I0331 03:58:14.716098 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 85.207131ms to process 1 records I0331 03:58:14.716112 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 86.645579ms to process 0 records I0331 03:58:14.716073 1 tasks_processing.go:74] worker 57 stopped. I0331 03:58:14.716119 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 86.394023ms to process 0 records I0331 03:58:14.716127 1 tasks_processing.go:74] worker 53 stopped. I0331 03:58:14.716135 1 tasks_processing.go:74] worker 40 stopped. I0331 03:58:14.716632 1 tasks_processing.go:74] worker 55 stopped. I0331 03:58:14.717250 1 recorder.go:75] Recording config/version with fingerprint=3c1e23e1d9e6d9f5f7fa40bad60a9d098988a9512037be0134ac28ba7982c8de I0331 03:58:14.717309 1 recorder.go:75] Recording config/id with fingerprint=9244ad45c719bb5bffd76fc90c63e7136d330b086b0af5ef6f58039e8e7829db I0331 03:58:14.717394 1 gather.go:177] gatherer "clusterconfig" function "version" took 84.820107ms to process 2 records I0331 03:58:14.718071 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=b28663d0f397da18085593d97b2bcfad4e7de002014a92552bd00c0975ff24c7 I0331 03:58:14.718092 1 tasks_processing.go:74] worker 23 stopped. I0331 03:58:14.718094 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=6829c5011d66f512a26d887be28565dd299c2f656707c568cd76eae086b64ff3 I0331 03:58:14.718110 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=942321de2ef71d8f0a1b7f21cc8946c1ecccf0643ce716e38f0154aefebb4ed7 I0331 03:58:14.718134 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=455b0f3ee73eb4b5d4b588d521243be8f99a9092c423302f3a857781d867ca27 I0331 03:58:14.718150 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=f351f58a9f07093fc83e741c8ae94aff7f4d906af194ce81155dee1656eaf9a0 I0331 03:58:14.718171 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=722b14ee98169d3163fb08636e29b70e04500492616cb271d885280ed5d0343b I0331 03:58:14.718183 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 86.737427ms to process 6 records I0331 03:58:14.718366 1 recorder.go:75] Recording config/authentication with fingerprint=a345b387eb237897f337dce1d60845b5515b58292df96b1ce8491104a3c4a0b0 I0331 03:58:14.718400 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 88.669865ms to process 1 records I0331 03:58:14.718417 1 tasks_processing.go:74] worker 36 stopped. I0331 03:58:14.718532 1 tasks_processing.go:74] worker 12 stopped. I0331 03:58:14.718734 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=1f5fb3ff87ac15b7f8adc2bf9efa1715b0ff6011d040b2b4c26a4f993b241d6f I0331 03:58:14.718752 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 89.322652ms to process 1 records I0331 03:58:14.718827 1 tasks_processing.go:74] worker 38 stopped. I0331 03:58:14.718998 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=e125863c64d14806e4100f8b5cd9e7b74a6315c7ceeeeed94a3065a0f7307fab I0331 03:58:14.719151 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=10554c0a094e1dc015e37b710f5d34f8d156a63def3b701cee2adae7b067c020 I0331 03:58:14.719161 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 90.381959ms to process 2 records I0331 03:58:14.719206 1 tasks_processing.go:74] worker 6 stopped. I0331 03:58:14.719275 1 recorder.go:75] Recording config/olm_operators with fingerprint=df08ca904b3055bb1838c08bb1ab7314fd14d8d64c7bc8b70b5d55b5e8d00d4a I0331 03:58:14.719293 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 91.132799ms to process 1 records W0331 03:58:14.719468 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0331 03:58:14.719530 1 tasks_processing.go:74] worker 7 stopped. I0331 03:58:14.719734 1 controller.go:212] Source scaController *sca.Controller is not ready I0331 03:58:14.719750 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0331 03:58:14.719755 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0331 03:58:14.719759 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0331 03:58:14.719763 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0331 03:58:14.719787 1 controller.go:489] The operator is still being initialized I0331 03:58:14.719798 1 controller.go:512] The operator is healthy I0331 03:58:14.719957 1 recorder.go:75] Recording config/node/ip-10-0-0-35.ec2.internal with fingerprint=9595e347ae6e797506aa4b61ca68bf8ba4450c5c8ea8c36d66d4e28c41a7e3c2 I0331 03:58:14.720060 1 recorder.go:75] Recording config/node/ip-10-0-2-201.ec2.internal with fingerprint=5003552e398ca414e2e8573b4c9e3f30d5b0c784fc1b3f82e776abcde8b3135a I0331 03:58:14.720079 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 91.356837ms to process 2 records I0331 03:58:14.720166 1 tasks_processing.go:74] worker 42 stopped. I0331 03:58:14.720520 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=76bc96f8b899bf996183d67ad8cc3ae0ded818122db1583e01e9503d29c2ff1e I0331 03:58:14.720536 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 89.264329ms to process 1 records I0331 03:58:14.720620 1 tasks_processing.go:74] worker 56 stopped. I0331 03:58:14.720645 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=1c4c78ebeedc7db1df6acfe3781d847ac7f9305d07f8a4128a58214e9e641c82 I0331 03:58:14.720678 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=3202b3602f3c2b30d953bf6e99e92a3648ccae30f4267a664a9634da46692472 I0331 03:58:14.720825 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=57c247cf54fa8f35b19557c448350adfea11e696f370aec226f3273ff911459f I0331 03:58:14.720836 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 89.526244ms to process 3 records I0331 03:58:14.721665 1 tasks_processing.go:74] worker 11 stopped. I0331 03:58:14.721681 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 92.183669ms to process 0 records I0331 03:58:14.723830 1 tasks_processing.go:74] worker 45 stopped. I0331 03:58:14.723901 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 93.26998ms to process 0 records I0331 03:58:14.724460 1 tasks_processing.go:74] worker 8 stopped. I0331 03:58:14.724491 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0331 03:58:14.724584 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 95.691321ms to process 1 records I0331 03:58:14.729577 1 tasks_processing.go:74] worker 30 stopped. I0331 03:58:14.730077 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=e3fa0c65533bbbef8b8ea1f4299c00096332153b25f9c44002fb4a09aaf89bad I0331 03:58:14.730291 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=7e4d8c0e34dc2207f878b4d0a4c0279ece7b05fa4dc0adf9b5f143045b3be139 I0331 03:58:14.730301 1 gather.go:177] gatherer "clusterconfig" function "crds" took 99.586214ms to process 2 records I0331 03:58:14.730312 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 98.996126ms to process 0 records I0331 03:58:14.730326 1 tasks_processing.go:74] worker 17 stopped. I0331 03:58:14.733164 1 tasks_processing.go:74] worker 48 stopped. I0331 03:58:14.733205 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0331 03:58:14.733218 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 102.360595ms to process 1 records I0331 03:58:14.734822 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0331 03:58:14.735112 1 tasks_processing.go:74] worker 10 stopped. I0331 03:58:14.736394 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-g4zhn with fingerprint=b5a832f7a480bc2541b1b9c99ef9de33da61a36b110753b39459608248b2d3d6 I0331 03:58:14.736492 1 recorder.go:75] Recording config/running_containers with fingerprint=db8f641f42edd95fa54674dc37428435ad91fc5e7b354aec786821988c996f58 I0331 03:58:14.736506 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 105.742218ms to process 2 records I0331 03:58:14.737068 1 tasks_processing.go:74] worker 39 stopped. I0331 03:58:14.737081 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 108.967315ms to process 0 records I0331 03:58:14.738749 1 base_controller.go:82] Caches are synced for LoggingSyncer I0331 03:58:14.738765 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0331 03:58:14.744978 1 tasks_processing.go:74] worker 31 stopped. E0331 03:58:14.744990 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0331 03:58:14.744995 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0331 03:58:14.744999 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0331 03:58:14.745008 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0331 03:58:14.745032 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0331 03:58:14.745036 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0331 03:58:14.745040 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0331 03:58:14.745049 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0331 03:58:14.745086 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0331 03:58:14.745094 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0331 03:58:14.745099 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 115.33449ms to process 7 records I0331 03:58:14.746504 1 tasks_processing.go:74] worker 43 stopped. E0331 03:58:14.746515 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0331 03:58:14.746520 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2pcf4goiisa5m55tcrhfco9v7iv9c0o4-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2pcf4goiisa5m55tcrhfco9v7iv9c0o4-primary-cert-bundle-secret" not found I0331 03:58:14.746559 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=52a053f8bc7075ff999eac79b6d898b5d1f40f8c1bf67150f08fd9d45efa85a2 I0331 03:58:14.746581 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 117.422851ms to process 1 records I0331 03:58:14.779979 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0331 03:58:14.783854 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0331 03:58:14.888325 1 sca.go:266] etc-pki-entitlement secret successfully created I0331 03:58:14.888341 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0331 03:58:14.928513 1 cluster_transfer.go:102] no available accepted cluster transfer I0331 03:58:14.928527 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer W0331 03:58:15.717677 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0331 03:58:16.717364 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0331 03:58:17.145971 1 tasks_processing.go:74] worker 35 stopped. I0331 03:58:17.146099 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=7c2025699cb08efa01561e2f21603b9a47ffd659df7ffcb484b68acbb6760036 I0331 03:58:17.146144 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=b1d3a57201f472fba248e6e22732d03f4ebac77f3d7204455295e9c6f31f376a I0331 03:58:17.146165 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=d4fdc2a4393ce64edd5b0a629b0785bcd2d380f214b96b15870a138e5665307d I0331 03:58:17.146175 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0331 03:58:17.146198 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=2cb82add28ef60409201e69710ab485d3248de2b7d6eaceee68ac6f902316748 I0331 03:58:17.146214 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0331 03:58:17.146235 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=b0d17d4bc7c5fee218eafd91fe240b1dc91c73b579c067f9a5fbb72bb2df76f2 I0331 03:58:17.146263 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=215f0bf211fb26e386d29e6fe5d9cb7a5414c53fb935dd243b64c7f7ca32877a I0331 03:58:17.146286 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=4d61f395a705f849ebe31efd4f2f737ed1a315d4102bf9440200795e2d57c385 I0331 03:58:17.146317 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=6179005c086e52d73c4f60adc4a95904c66a09b9b5ed0324d1494aba19f74320 I0331 03:58:17.146599 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=b9839543657e010a0e404049a4ce1ca97e388d495b8485016e14f3b96a0b53fb I0331 03:58:17.146620 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=ad7a5ae04bc4f8e0eec3ae8dbed4e754b9c2dd7e27c76e6f077e2350ed8db6e3 I0331 03:58:17.146629 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0331 03:58:17.146646 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=13f3a28bf6957bde3edba4cef997196160bf08e2a66cad4bc69e8b9740ff4752 I0331 03:58:17.146656 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0331 03:58:17.146673 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=59070e5b30ecc3a5dd8206e8e95e26685fe2be64c5f00d8ce18b115a3840e0ec I0331 03:58:17.146682 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0331 03:58:17.146701 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=ee915a490289986c380245e46542895aa1fca8133bcf094be81de9c71e42b5f7 I0331 03:58:17.146710 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0331 03:58:17.146730 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=176ffc1303bc4ef5f90f8ebf5a527263cb7ea06de8fb1307e6a01eede32161c4 I0331 03:58:17.146830 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=3647b52e3c63041b21250bc7ca981883750a1ffe219d173e7f7818f89090cd53 I0331 03:58:17.146839 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0331 03:58:17.146853 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0331 03:58:17.146874 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0331 03:58:17.146896 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=7724179457d63f840098725a5301cfab1320325a61a08bb2b85b0c4ad5f935ea I0331 03:58:17.146928 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=776f934bded04aeaa535f2b03a435c31f2a1bbe6dc6dc2352a1b2cdeefacc18f I0331 03:58:17.146938 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0331 03:58:17.146955 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=3d091d93fc024e959b487aabca4f69beadc966e610f6d8c2919d85057ac103f2 I0331 03:58:17.146965 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0331 03:58:17.146982 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=d84baeb3242d1e693bc3b28e6cb2addf8c5388e05bc8eb1f72d1202e3805aab0 I0331 03:58:17.146993 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0331 03:58:17.147005 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=92328d8231f25710698192f70a455b4c584233a0b3979bacadbc3717a579beda I0331 03:58:17.147021 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=97d1d1bd4d9258caeffe219b41211d58ab38cd9b5d6dc0082413967d3b1c55ad I0331 03:58:17.147043 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=85b89a38b0731aab89559115368196a5009e19e35433c6ddd460503df065ed21 I0331 03:58:17.147065 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=b97a5eb5a8491ae5c8bb98a25ad7f0b93f80b35068273050139fcef8f5148a5a I0331 03:58:17.147074 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0331 03:58:17.147098 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=a3e9cfb4b21cd73f5add3803d1d5969d803a2b23c54bc0bf754ae616c2d0a75d I0331 03:58:17.147115 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0331 03:58:17.147124 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0331 03:58:17.147132 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.517504669s to process 39 records W0331 03:58:17.717922 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0331 03:58:18.717023 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0331 03:58:19.717798 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0331 03:58:19.717829 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0331 03:58:19.717848 1 tasks_processing.go:74] worker 24 stopped. E0331 03:58:19.717861 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0331 03:58:19.717878 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0331 03:58:19.717893 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.089556857s to process 1 records I0331 03:58:27.529146 1 tasks_processing.go:74] worker 25 stopped. I0331 03:58:27.529190 1 recorder.go:75] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0331 03:58:27.529202 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.899067925s to process 1 records I0331 03:58:28.236575 1 tasks_processing.go:74] worker 22 stopped. I0331 03:58:28.236885 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=04b1dfc65b82e4ffa7a0e08c48682e6aa2f3ebd208f3909e4e17b361a7606d8c I0331 03:58:28.236901 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.60696019s to process 1 records E0331 03:58:28.236970 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.609s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machines\" failed with an error, function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"active_alerts\" failed with an error, function \"silenced_alerts\" failed with an error, function \"support_secret\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0331 03:58:28.238074 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machines" failed with an error, function "tsdb_status" failed with an error, function "metrics" failed with an error, function "active_alerts" failed with an error, function "silenced_alerts" failed with an error, function "support_secret" failed with an error, function "machine_healthchecks" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0331 03:58:28.238089 1 periodic.go:209] Running workloads gatherer I0331 03:58:28.238104 1 tasks_processing.go:45] number of workers: 2 I0331 03:58:28.238109 1 tasks_processing.go:69] worker 1 listening for tasks. I0331 03:58:28.238114 1 tasks_processing.go:71] worker 1 working on workload_info task. I0331 03:58:28.238177 1 tasks_processing.go:69] worker 0 listening for tasks. I0331 03:58:28.238252 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0331 03:58:28.244719 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-35.ec2.internal... I0331 03:58:28.244725 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-201.ec2.internal... I0331 03:58:28.263916 1 tasks_processing.go:74] worker 0 stopped. I0331 03:58:28.263935 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 25.642358ms to process 0 records I0331 03:58:33.412688 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 5s I0331 03:58:33.435175 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 23s for image data I0331 03:58:33.443522 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (9ms) I0331 03:58:33.451997 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (8ms) I0331 03:58:33.460453 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (8ms) I0331 03:58:33.468798 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (8ms) I0331 03:58:33.476718 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (8ms) I0331 03:58:33.484954 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (8ms) I0331 03:58:33.492738 1 gather_workloads_info.go:387] No image sha256:90bdf9e678fe18e35401e34e1cb0912d822e9cb32b9e41cc545870540e9a87f2 (8ms) I0331 03:58:33.501364 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (9ms) I0331 03:58:33.509216 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (8ms) I0331 03:58:33.517401 1 gather_workloads_info.go:387] No image sha256:513b1166e376feca472fdd5174f6abf0abf02925d6a21b7b4d89a2ff2fc67bed (8ms) I0331 03:58:33.543104 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (26ms) I0331 03:58:33.646035 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (103ms) I0331 03:58:33.745040 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (99ms) I0331 03:58:33.843371 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (98ms) I0331 03:58:33.944435 1 gather_workloads_info.go:387] No image sha256:1477f3287cc75352305274f248f781c1b7c2a8696f4f7bc450c45a4a5f5b10ea (101ms) I0331 03:58:34.045456 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (101ms) I0331 03:58:34.144977 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (100ms) I0331 03:58:34.244803 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (100ms) I0331 03:58:34.344857 1 gather_workloads_info.go:387] No image sha256:1e65f9f5490fbc930d2dce96f03ddcb28990b839d7dc011ef5de5bec6c5095bf (100ms) I0331 03:58:34.444436 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (100ms) I0331 03:58:34.551237 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (107ms) I0331 03:58:34.644558 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (93ms) I0331 03:58:34.744632 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (100ms) I0331 03:58:34.844321 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0331 03:58:34.943945 1 gather_workloads_info.go:398] Found image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (100ms) I0331 03:58:35.044519 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0331 03:58:35.144532 1 gather_workloads_info.go:387] No image sha256:0af73ad7a47c9ff53269ae2bc5163bf9e389085220c34e4bd42c92c4e7a85f6d (100ms) I0331 03:58:35.244782 1 gather_workloads_info.go:387] No image sha256:0092c6ff139f8698c91349d507eb3182bc5d6c4ae040f672e347ff504accdea2 (100ms) I0331 03:58:35.345041 1 gather_workloads_info.go:398] Found image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (100ms) I0331 03:58:35.443899 1 gather_workloads_info.go:387] No image sha256:74713675d580a47a56b628469c4c05c41e114e0b935960d718ff0cf7e72f613d (99ms) I0331 03:58:35.544820 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (101ms) I0331 03:58:35.644272 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (99ms) I0331 03:58:35.743803 1 gather_workloads_info.go:387] No image sha256:101993f6e65d6fda8b90c583b718b720809182961f69bc777295f47e6df4d7ce (100ms) I0331 03:58:35.845688 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (102ms) I0331 03:58:35.944391 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (99ms) I0331 03:58:36.043998 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (100ms) I0331 03:58:36.144282 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (100ms) I0331 03:58:36.243951 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0331 03:58:36.346543 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (103ms) I0331 03:58:36.443899 1 gather_workloads_info.go:387] No image sha256:b99613494e9d6a4a8207323872d63cc02b830fc80a8bf972e62c64381aa4cd17 (97ms) I0331 03:58:36.544996 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (101ms) I0331 03:58:36.645413 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (100ms) I0331 03:58:36.744767 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (99ms) I0331 03:58:36.845506 1 gather_workloads_info.go:387] No image sha256:6e5df2aab9ce759e8f957341a11c087c233307e3c8f8760d21270efc19ccd3ce (101ms) I0331 03:58:36.945588 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0331 03:58:37.045987 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (100ms) I0331 03:58:37.144612 1 gather_workloads_info.go:387] No image sha256:7bd3e88d6d3f878a89203df590b684c5a77bf08df4717451d3197ab74683a80e (99ms) I0331 03:58:37.249685 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (105ms) I0331 03:58:37.249718 1 tasks_processing.go:74] worker 1 stopped. E0331 03:58:37.249727 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: received unexpected status code 500 Internal Server Error from https://10.129.0.12:8443/gather_runtime_info I0331 03:58:37.250036 1 recorder.go:75] Recording config/workload_info with fingerprint=0a3e015c39ee8aefb819d093d46f390e7710db4be3ef17af8a57d4c006db406d I0331 03:58:37.250050 1 gather.go:177] gatherer "workloads" function "workload_info" took 9.011596721s to process 1 records E0331 03:58:37.250076 1 periodic.go:247] "Unhandled Error" err="workloads failed after 9.011s with: function \"workload_info\" failed with an error" I0331 03:58:37.251172 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0331 03:58:37.251184 1 periodic.go:209] Running conditional gatherer I0331 03:58:37.256929 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0331 03:58:37.421443 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer E0331 03:58:37.423635 1 conditional_gatherer.go:324] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp 172.30.180.214:9091: connect: connection refused I0331 03:58:37.423704 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0331 03:58:37.430111 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0331 03:58:37.430127 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:58:37.430132 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:58:37.430136 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:58:37.430140 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:58:37.430144 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:58:37.430147 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:58:37.430150 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:58:37.430153 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:58:37.430156 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:58:37.430553 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0331 03:58:37.430579 1 tasks_processing.go:45] number of workers: 3 I0331 03:58:37.430589 1 tasks_processing.go:69] worker 2 listening for tasks. I0331 03:58:37.430595 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0331 03:58:37.430686 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0331 03:58:37.430704 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.422µs to process 1 records I0331 03:58:37.430713 1 tasks_processing.go:69] worker 0 listening for tasks. I0331 03:58:37.430726 1 tasks_processing.go:69] worker 1 listening for tasks. I0331 03:58:37.430734 1 tasks_processing.go:74] worker 1 stopped. I0331 03:58:37.430740 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0331 03:58:37.430746 1 tasks_processing.go:74] worker 2 stopped. I0331 03:58:37.430862 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=9fafdcbf373f1f367f1d69551ea59114592cce1681189be0f3dde774e0530661 I0331 03:58:37.430877 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.013µs to process 1 records I0331 03:58:37.430886 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0331 03:58:37.431364 1 tasks_processing.go:74] worker 0 stopped. I0331 03:58:37.431466 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 463.994µs to process 0 records I0331 03:58:37.431537 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0331 03:58:37.431580 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0331 03:58:37.470112 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=a87c13c90c32e6f78857b46b79027736dce3dbf1dbc3ba79a033054f2bff9cef I0331 03:58:37.470259 1 diskrecorder.go:70] Writing 103 records to /var/lib/insights-operator/insights-2026-03-31-035837.tar.gz I0331 03:58:37.477348 1 diskrecorder.go:51] Wrote 103 records to disk in 7ms I0331 03:58:37.477391 1 periodic.go:278] Gathering cluster info every 2h0m0s I0331 03:58:37.477405 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0331 03:59:44.628572 1 diskrecorder.go:223] Found files to send: insights-2026-03-31-035837.tar.gz I0331 03:59:44.628601 1 insightsuploader.go:150] Checking archives to upload periodically every 15m53.588612227s I0331 03:59:44.628610 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0331 03:59:44.638067 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0331 03:59:44.906498 1 requests.go:87] Successfully reported id=2026-03-31T03:59:44Z x-rh-insights-request-id=af31ea1553054f8fbc8b65cc5b3c7688, wrote=46740 I0331 03:59:44.906552 1 insightsuploader.go:187] Uploaded report successfully in 277.934415ms I0331 03:59:44.906578 1 controller.go:128] Initializing last reported time to 2026-03-31T03:59:44Z I0331 03:59:44.906597 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0331 03:59:44.906612 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0331 03:59:44.906620 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0331 03:59:44.913177 1 controller.go:512] The operator is healthy I0331 04:00:14.647177 1 controller.go:512] The operator is healthy I0331 04:00:47.871910 1 insightsreport.go:137] Pulling report from smart-proxy I0331 04:00:47.871938 1 insightsreport.go:149] Retrieving report I0331 04:00:47.879485 1 requests.go:110] Retrieving report for cluster: 20273850-a2b6-4bdc-a2bd-d7bf471f4f58 I0331 04:00:47.879495 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/20273850-a2b6-4bdc-a2bd-d7bf471f4f58/reports I0331 04:00:47.882082 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/20273850-a2b6-4bdc-a2bd-d7bf471f4f58/reports I0331 04:00:48.077993 1 insightsreport.go:184] Report retrieved I0331 04:00:48.104236 1 insightsreport.go:239] Report retrieved correctly I0331 04:02:14.646881 1 controller.go:512] The operator is healthy I0331 04:03:14.617759 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0331 04:03:14.623546 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0331 04:03:14.623566 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0331 04:03:14.627785 1 secretconfigobserver.go:119] support secret does not exist I0331 04:04:14.646817 1 controller.go:512] The operator is healthy I0331 04:06:14.647723 1 controller.go:512] The operator is healthy I0331 04:08:14.614467 1 configmapobserver.go:84] configmaps "insights-config" not found I0331 04:08:14.628490 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0331 04:08:14.633086 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0331 04:08:14.633102 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0331 04:08:14.637186 1 secretconfigobserver.go:119] support secret does not exist I0331 04:08:14.644285 1 controller.go:512] The operator is healthy I0331 04:10:14.648364 1 controller.go:512] The operator is healthy I0331 04:12:14.646690 1 controller.go:512] The operator is healthy I0331 04:13:14.637922 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0331 04:13:14.643371 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0331 04:13:14.643403 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0331 04:13:14.647621 1 secretconfigobserver.go:119] support secret does not exist I0331 04:14:14.645530 1 controller.go:512] The operator is healthy I0331 04:15:43.497213 1 insightsuploader.go:146] Nothing to report since 2026-03-31T03:59:44Z I0331 04:16:14.647787 1 controller.go:512] The operator is healthy I0331 04:18:14.614983 1 configmapobserver.go:84] configmaps "insights-config" not found I0331 04:18:14.619021 1 configmapobserver.go:84] configmaps "insights-config" not found I0331 04:18:14.644318 1 controller.go:512] The operator is healthy I0331 04:18:14.648179 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0331 04:18:14.652541 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0331 04:18:14.652555 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0331 04:18:14.656491 1 secretconfigobserver.go:119] support secret does not exist I0331 04:20:14.646083 1 controller.go:512] The operator is healthy I0331 04:22:14.646435 1 controller.go:512] The operator is healthy I0331 04:23:14.656833 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0331 04:23:14.662901 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0331 04:23:14.662919 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0331 04:23:14.667349 1 secretconfigobserver.go:119] support secret does not exist I0331 04:24:14.645606 1 controller.go:512] The operator is healthy I0331 04:26:14.646079 1 controller.go:512] The operator is healthy I0331 04:28:14.614800 1 configmapobserver.go:84] configmaps "insights-config" not found I0331 04:28:14.644695 1 controller.go:512] The operator is healthy I0331 04:28:14.667442 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0331 04:28:14.672537 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0331 04:28:14.672556 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0331 04:28:14.678806 1 secretconfigobserver.go:119] support secret does not exist I0331 04:30:14.645466 1 controller.go:512] The operator is healthy I0331 04:31:22.059422 1 insightsuploader.go:146] Nothing to report since 2026-03-31T03:59:44Z I0331 04:32:14.646130 1 controller.go:512] The operator is healthy