I0418 07:53:56.296026 1 cmd.go:253] Using service-serving-cert provided certificates I0418 07:53:56.296528 1 observer_polling.go:159] Starting file observer I0418 07:53:56.532062 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0418 07:53:56.532325 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0418 07:53:56.533272 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0418 07:53:56.533294 1 secure_serving.go:57] Forcing use of http/1.1 only W0418 07:53:56.533316 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0418 07:53:56.533323 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0418 07:53:56.533331 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0418 07:53:56.533335 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0418 07:53:56.533339 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0418 07:53:56.533343 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0418 07:53:56.537255 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0418 07:53:56.537273 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"ef03ad02-2bf8-4594-8ccc-78f2cf12f204", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0418 07:53:56.539737 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0418 07:53:56.539749 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0418 07:53:56.539759 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0418 07:53:56.539760 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0418 07:53:56.539813 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0418 07:53:56.539799 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0418 07:53:56.540125 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0418 07:53:56.540413 1 secure_serving.go:213] Serving securely on [::]:8443 I0418 07:53:56.540433 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0418 07:53:56.545352 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0418 07:53:56.545375 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0418 07:53:56.545474 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0418 07:53:56.551301 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0418 07:53:56.551318 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0418 07:53:56.557424 1 secretconfigobserver.go:119] support secret does not exist I0418 07:53:56.561639 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0418 07:53:56.566298 1 secretconfigobserver.go:119] support secret does not exist I0418 07:53:56.567796 1 recorder.go:161] Pruning old reports every 6h17m4s, max age is 288h0m0s I0418 07:53:56.572463 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0418 07:53:56.572470 1 periodic.go:209] Running clusterconfig gatherer I0418 07:53:56.572478 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0418 07:53:56.572502 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0418 07:53:56.572506 1 insightsreport.go:296] Starting report retriever I0418 07:53:56.572512 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0418 07:53:56.572514 1 tasks_processing.go:45] number of workers: 64 I0418 07:53:56.572532 1 tasks_processing.go:69] worker 0 listening for tasks. I0418 07:53:56.572534 1 tasks_processing.go:69] worker 2 listening for tasks. I0418 07:53:56.572540 1 tasks_processing.go:69] worker 1 listening for tasks. I0418 07:53:56.572544 1 tasks_processing.go:69] worker 19 listening for tasks. I0418 07:53:56.572545 1 tasks_processing.go:69] worker 8 listening for tasks. I0418 07:53:56.572552 1 tasks_processing.go:69] worker 9 listening for tasks. I0418 07:53:56.572554 1 tasks_processing.go:69] worker 18 listening for tasks. I0418 07:53:56.572558 1 tasks_processing.go:71] worker 9 working on infrastructures task. I0418 07:53:56.572560 1 tasks_processing.go:69] worker 3 listening for tasks. I0418 07:53:56.572562 1 tasks_processing.go:69] worker 11 listening for tasks. I0418 07:53:56.572564 1 tasks_processing.go:69] worker 20 listening for tasks. I0418 07:53:56.572568 1 tasks_processing.go:69] worker 44 listening for tasks. I0418 07:53:56.572568 1 tasks_processing.go:69] worker 16 listening for tasks. I0418 07:53:56.572576 1 tasks_processing.go:69] worker 21 listening for tasks. I0418 07:53:56.572576 1 tasks_processing.go:69] worker 5 listening for tasks. I0418 07:53:56.572577 1 tasks_processing.go:69] worker 10 listening for tasks. I0418 07:53:56.572585 1 tasks_processing.go:69] worker 59 listening for tasks. I0418 07:53:56.572569 1 tasks_processing.go:69] worker 4 listening for tasks. I0418 07:53:56.572589 1 tasks_processing.go:69] worker 14 listening for tasks. I0418 07:53:56.572550 1 tasks_processing.go:69] worker 17 listening for tasks. I0418 07:53:56.572595 1 tasks_processing.go:69] worker 7 listening for tasks. I0418 07:53:56.572599 1 tasks_processing.go:69] worker 15 listening for tasks. I0418 07:53:56.572572 1 tasks_processing.go:69] worker 45 listening for tasks. I0418 07:53:56.572602 1 tasks_processing.go:71] worker 19 working on machine_configs task. I0418 07:53:56.572606 1 tasks_processing.go:69] worker 36 listening for tasks. I0418 07:53:56.572609 1 tasks_processing.go:69] worker 50 listening for tasks. I0418 07:53:56.572611 1 tasks_processing.go:69] worker 12 listening for tasks. I0418 07:53:56.572613 1 tasks_processing.go:69] worker 37 listening for tasks. I0418 07:53:56.572609 1 tasks_processing.go:69] worker 51 listening for tasks. I0418 07:53:56.572576 1 tasks_processing.go:69] worker 57 listening for tasks. I0418 07:53:56.572619 1 tasks_processing.go:69] worker 26 listening for tasks. I0418 07:53:56.572548 1 tasks_processing.go:69] worker 13 listening for tasks. I0418 07:53:56.573257 1 tasks_processing.go:69] worker 63 listening for tasks. I0418 07:53:56.572751 1 tasks_processing.go:69] worker 27 listening for tasks. I0418 07:53:56.573278 1 tasks_processing.go:71] worker 63 working on container_images task. I0418 07:53:56.573378 1 tasks_processing.go:69] worker 41 listening for tasks. I0418 07:53:56.573481 1 tasks_processing.go:69] worker 31 listening for tasks. I0418 07:53:56.573545 1 tasks_processing.go:69] worker 29 listening for tasks. I0418 07:53:56.573598 1 tasks_processing.go:69] worker 30 listening for tasks. I0418 07:53:56.572581 1 tasks_processing.go:69] worker 22 listening for tasks. I0418 07:53:56.572588 1 tasks_processing.go:69] worker 6 listening for tasks. I0418 07:53:56.572591 1 tasks_processing.go:69] worker 60 listening for tasks. I0418 07:53:56.572591 1 tasks_processing.go:69] worker 23 listening for tasks. I0418 07:53:56.572594 1 tasks_processing.go:71] worker 2 working on metrics task. I0418 07:53:56.573649 1 tasks_processing.go:69] worker 61 listening for tasks. I0418 07:53:56.572596 1 tasks_processing.go:69] worker 24 listening for tasks. I0418 07:53:56.573660 1 tasks_processing.go:69] worker 28 listening for tasks. I0418 07:53:56.572558 1 tasks_processing.go:71] worker 8 working on support_secret task. I0418 07:53:56.572598 1 tasks_processing.go:71] worker 1 working on sap_pods task. I0418 07:53:56.572599 1 tasks_processing.go:69] worker 56 listening for tasks. I0418 07:53:56.572601 1 tasks_processing.go:69] worker 35 listening for tasks. I0418 07:53:56.572605 1 tasks_processing.go:69] worker 34 listening for tasks. I0418 07:53:56.572611 1 tasks_processing.go:69] worker 25 listening for tasks. I0418 07:53:56.572619 1 tasks_processing.go:69] worker 46 listening for tasks. I0418 07:53:56.572619 1 tasks_processing.go:69] worker 54 listening for tasks. I0418 07:53:56.572622 1 tasks_processing.go:69] worker 38 listening for tasks. I0418 07:53:56.572626 1 tasks_processing.go:69] worker 47 listening for tasks. I0418 07:53:56.572626 1 tasks_processing.go:69] worker 55 listening for tasks. I0418 07:53:56.572608 1 tasks_processing.go:71] worker 0 working on openshift_logging task. I0418 07:53:56.573738 1 tasks_processing.go:71] worker 11 working on feature_gates task. I0418 07:53:56.573743 1 tasks_processing.go:71] worker 3 working on openstack_controlplanes task. I0418 07:53:56.573758 1 tasks_processing.go:69] worker 62 listening for tasks. I0418 07:53:56.573758 1 tasks_processing.go:71] worker 18 working on machine_config_pools task. I0418 07:53:56.573786 1 tasks_processing.go:71] worker 20 working on machine_autoscalers task. I0418 07:53:56.573827 1 tasks_processing.go:69] worker 53 listening for tasks. I0418 07:53:56.573843 1 tasks_processing.go:71] worker 26 working on sap_datahubs task. I0418 07:53:56.573929 1 tasks_processing.go:71] worker 44 working on cluster_apiserver task. I0418 07:53:56.572804 1 tasks_processing.go:69] worker 39 listening for tasks. I0418 07:53:56.572885 1 tasks_processing.go:69] worker 52 listening for tasks. I0418 07:53:56.573086 1 tasks_processing.go:69] worker 48 listening for tasks. I0418 07:53:56.573956 1 tasks_processing.go:71] worker 4 working on monitoring_persistent_volumes task. I0418 07:53:56.573749 1 tasks_processing.go:71] worker 21 working on machines task. I0418 07:53:56.573874 1 tasks_processing.go:71] worker 14 working on olm_operators task. I0418 07:53:56.574006 1 tasks_processing.go:71] worker 15 working on openstack_dataplanedeployments task. I0418 07:53:56.574049 1 tasks_processing.go:71] worker 36 working on storage_cluster task. I0418 07:53:56.573654 1 tasks_processing.go:69] worker 33 listening for tasks. I0418 07:53:56.574059 1 tasks_processing.go:71] worker 17 working on nodenetworkstates task. I0418 07:53:56.574067 1 tasks_processing.go:71] worker 7 working on operators task. I0418 07:53:56.573673 1 tasks_processing.go:69] worker 42 listening for tasks. I0418 07:53:56.573671 1 tasks_processing.go:69] worker 32 listening for tasks. I0418 07:53:56.573993 1 tasks_processing.go:71] worker 5 working on operators_pods_and_events task. I0418 07:53:56.574121 1 tasks_processing.go:71] worker 10 working on image_registries task. I0418 07:53:56.574156 1 tasks_processing.go:71] worker 16 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0418 07:53:56.574169 1 tasks_processing.go:71] worker 57 working on image_pruners task. I0418 07:53:56.574192 1 tasks_processing.go:71] worker 59 working on overlapping_namespace_uids task. I0418 07:53:56.574204 1 tasks_processing.go:71] worker 45 working on version task. I0418 07:53:56.574207 1 tasks_processing.go:71] worker 37 working on openstack_version task. I0418 07:53:56.574237 1 tasks_processing.go:69] worker 49 listening for tasks. I0418 07:53:56.574427 1 tasks_processing.go:71] worker 49 working on crds task. I0418 07:53:56.574476 1 tasks_processing.go:71] worker 27 working on install_plans task. I0418 07:53:56.574499 1 tasks_processing.go:69] worker 43 listening for tasks. I0418 07:53:56.574518 1 tasks_processing.go:71] worker 43 working on clusterroles task. I0418 07:53:56.574520 1 tasks_processing.go:71] worker 41 working on image task. I0418 07:53:56.574565 1 tasks_processing.go:71] worker 53 working on node_logs task. I0418 07:53:56.574574 1 tasks_processing.go:71] worker 48 working on nodenetworkconfigurationpolicies task. I0418 07:53:56.574577 1 tasks_processing.go:71] worker 31 working on openshift_machine_api_events task. I0418 07:53:56.574159 1 tasks_processing.go:71] worker 51 working on config_maps task. I0418 07:53:56.574583 1 tasks_processing.go:71] worker 29 working on ingress_certificates task. I0418 07:53:56.574589 1 tasks_processing.go:71] worker 52 working on authentication task. I0418 07:53:56.574597 1 tasks_processing.go:71] worker 42 working on service_accounts task. I0418 07:53:56.574609 1 tasks_processing.go:71] worker 32 working on storage_classes task. I0418 07:53:56.574614 1 tasks_processing.go:71] worker 28 working on machine_sets task. I0418 07:53:56.574615 1 tasks_processing.go:71] worker 61 working on certificate_signing_requests task. I0418 07:53:56.574618 1 tasks_processing.go:71] worker 30 working on tsdb_status task. I0418 07:53:56.574621 1 tasks_processing.go:71] worker 62 working on active_alerts task. I0418 07:53:56.574585 1 tasks_processing.go:71] worker 39 working on oauths task. I0418 07:53:56.574580 1 tasks_processing.go:71] worker 33 working on lokistack task. I0418 07:53:56.574761 1 tasks_processing.go:71] worker 22 working on nodes task. I0418 07:53:56.574823 1 tasks_processing.go:71] worker 35 working on dvo_metrics task. I0418 07:53:56.572581 1 tasks_processing.go:69] worker 58 listening for tasks. I0418 07:53:56.574916 1 tasks_processing.go:69] worker 40 listening for tasks. I0418 07:53:56.574925 1 tasks_processing.go:71] worker 40 working on container_runtime_configs task. I0418 07:53:56.575558 1 tasks_processing.go:71] worker 56 working on aggregated_monitoring_cr_names task. I0418 07:53:56.575631 1 tasks_processing.go:71] worker 34 working on pdbs task. I0418 07:53:56.575637 1 tasks_processing.go:71] worker 38 working on proxies task. I0418 07:53:56.575649 1 tasks_processing.go:71] worker 55 working on schedulers task. I0418 07:53:56.575701 1 tasks_processing.go:71] worker 54 working on validating_webhook_configurations task. I0418 07:53:56.575761 1 tasks_processing.go:71] worker 60 working on ceph_cluster task. I0418 07:53:56.575817 1 tasks_processing.go:71] worker 24 working on cost_management_metrics_configs task. I0418 07:53:56.575871 1 tasks_processing.go:71] worker 12 working on silenced_alerts task. I0418 07:53:56.575846 1 tasks_processing.go:71] worker 25 working on machine_healthchecks task. I0418 07:53:56.575925 1 tasks_processing.go:71] worker 46 working on qemu_kubevirt_launcher_logs task. I0418 07:53:56.575720 1 tasks_processing.go:71] worker 23 working on pod_network_connectivity_checks task. I0418 07:53:56.576184 1 tasks_processing.go:71] worker 58 working on ingress task. I0418 07:53:56.575852 1 tasks_processing.go:71] worker 47 working on jaegers task. I0418 07:53:56.575715 1 tasks_processing.go:71] worker 13 working on sap_config task. I0418 07:53:56.576231 1 tasks_processing.go:71] worker 6 working on mutating_webhook_configurations task. I0418 07:53:56.575856 1 tasks_processing.go:71] worker 50 working on openstack_dataplanenodesets task. I0418 07:53:56.578587 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0418 07:53:56.578605 1 controller.go:326] The initial operator extension status is healthy I0418 07:53:56.578612 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0418 07:53:56.578617 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0418 07:53:56.578619 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0418 07:53:56.578637 1 controller.go:489] The operator is still being initialized I0418 07:53:56.578644 1 controller.go:512] The operator is healthy I0418 07:53:56.578703 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0418 07:53:56.578729 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0418 07:53:56.578840 1 operator.go:288] started I0418 07:53:56.578860 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer E0418 07:53:56.580289 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.4.197:9091: connect: connection refused I0418 07:53:56.580306 1 tasks_processing.go:71] worker 30 working on networks task. E0418 07:53:56.580323 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.4.197:9091: connect: connection refused E0418 07:53:56.580331 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.4.197:9091: connect: connection refused I0418 07:53:56.580344 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 5.674696ms to process 0 records E0418 07:53:56.580357 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.4.197:9091: connect: connection refused I0418 07:53:56.580372 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 6.688225ms to process 0 records I0418 07:53:56.580385 1 tasks_processing.go:74] worker 2 stopped. I0418 07:53:56.582928 1 tasks_processing.go:74] worker 3 stopped. I0418 07:53:56.582941 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 9.170523ms to process 0 records I0418 07:53:56.582948 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 8.71986ms to process 0 records I0418 07:53:56.582953 1 tasks_processing.go:74] worker 37 stopped. I0418 07:53:56.591596 1 tasks_processing.go:74] worker 0 stopped. I0418 07:53:56.591611 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 17.884593ms to process 0 records I0418 07:53:56.591625 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 17.818316ms to process 0 records I0418 07:53:56.591630 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 17.550826ms to process 0 records I0418 07:53:56.591634 1 tasks_processing.go:74] worker 17 stopped. I0418 07:53:56.591638 1 tasks_processing.go:74] worker 20 stopped. I0418 07:53:56.601773 1 tasks_processing.go:74] worker 33 stopped. I0418 07:53:56.601784 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 27.063583ms to process 0 records I0418 07:53:56.601925 1 tasks_processing.go:74] worker 48 stopped. I0418 07:53:56.601933 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 27.343488ms to process 0 records I0418 07:53:56.601962 1 tasks_processing.go:74] worker 28 stopped. I0418 07:53:56.601979 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 27.338826ms to process 0 records I0418 07:53:56.602109 1 tasks_processing.go:74] worker 9 stopped. I0418 07:53:56.602937 1 recorder.go:75] Recording config/infrastructure with fingerprint=b72abaff83667f113fade7f044ee1fb911edfbd9f1c5f83ea61a3fda0b8f38ec I0418 07:53:56.602958 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 29.415854ms to process 1 records I0418 07:53:56.602965 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 26.173021ms to process 0 records I0418 07:53:56.602993 1 tasks_processing.go:74] worker 11 stopped. I0418 07:53:56.603006 1 tasks_processing.go:74] worker 60 stopped. I0418 07:53:56.603516 1 recorder.go:75] Recording config/featuregate with fingerprint=3a8ba3e34078be5cb1a2093a9d55059b2cb314b381a7336b3cb3cabb8d2e875c I0418 07:53:56.603543 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 28.279226ms to process 1 records I0418 07:53:56.603680 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0418 07:53:56.603694 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0418 07:53:56.603700 1 controller.go:212] Source scaController *sca.Controller is not ready I0418 07:53:56.603711 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0418 07:53:56.603715 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0418 07:53:56.603796 1 controller.go:489] The operator is still being initialized I0418 07:53:56.603809 1 controller.go:512] The operator is healthy I0418 07:53:56.603815 1 recorder.go:75] Recording config/apiserver with fingerprint=fb69d2185bf2a126f825986e83d80158d0a9bbd6d72859eef2014eb0fbff3a4d I0418 07:53:56.603826 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 28.237683ms to process 1 records I0418 07:53:56.603834 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 26.355574ms to process 0 records I0418 07:53:56.603847 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 28.477663ms to process 0 records I0418 07:53:56.603853 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 28.402359ms to process 0 records E0418 07:53:56.603861 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0418 07:53:56.603872 1 gather.go:177] gatherer "clusterconfig" function "machines" took 28.324921ms to process 0 records I0418 07:53:56.603887 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 26.184839ms to process 0 records E0418 07:53:56.603920 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0418 07:53:56.603931 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 26.291686ms to process 0 records I0418 07:53:56.603933 1 tasks_processing.go:74] worker 18 stopped. E0418 07:53:56.603939 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0418 07:53:56.603943 1 tasks_processing.go:74] worker 44 stopped. I0418 07:53:56.603944 1 tasks_processing.go:74] worker 21 stopped. I0418 07:53:56.603949 1 tasks_processing.go:74] worker 24 stopped. I0418 07:53:56.603950 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 26.248913ms to process 0 records I0418 07:53:56.603957 1 tasks_processing.go:74] worker 23 stopped. I0418 07:53:56.603960 1 tasks_processing.go:74] worker 26 stopped. I0418 07:53:56.603964 1 tasks_processing.go:74] worker 47 stopped. I0418 07:53:56.603969 1 tasks_processing.go:74] worker 25 stopped. I0418 07:53:56.604115 1 tasks_processing.go:74] worker 57 stopped. I0418 07:53:56.604948 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=bc5837b5578fdad20a8b8a29c1516389fd7561a3fad87abad55b73e9b3b7c5ce I0418 07:53:56.604965 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 28.861403ms to process 1 records I0418 07:53:56.604972 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 28.895591ms to process 0 records I0418 07:53:56.604975 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 29.865225ms to process 0 records I0418 07:53:56.604979 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 29.364304ms to process 0 records I0418 07:53:56.604980 1 tasks_processing.go:74] worker 40 stopped. I0418 07:53:56.604986 1 tasks_processing.go:74] worker 4 stopped. I0418 07:53:56.604982 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 30.433634ms to process 0 records I0418 07:53:56.604992 1 tasks_processing.go:74] worker 36 stopped. I0418 07:53:56.605002 1 tasks_processing.go:74] worker 53 stopped. I0418 07:53:56.605058 1 tasks_processing.go:74] worker 32 stopped. I0418 07:53:56.605132 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=6b02df5f96ff51640271e17b479b1e836d42aaec6c18a970da09e953694cf213 I0418 07:53:56.605160 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=86920e872fd5b98ce132b9170a12eb25ee06699eaa245ad1622160f6342898ae I0418 07:53:56.605173 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 29.802851ms to process 2 records E0418 07:53:56.605188 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0418 07:53:56.605203 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 31.159137ms to process 0 records I0418 07:53:56.605252 1 gather_logs.go:145] no pods in namespace were found I0418 07:53:56.605267 1 tasks_processing.go:74] worker 8 stopped. I0418 07:53:56.605334 1 recorder.go:75] Recording config/ingress with fingerprint=41320043b4a8b57f7b740b8ad3fbe8182505147d28027681001851ab117139d9 I0418 07:53:56.605336 1 tasks_processing.go:74] worker 58 stopped. I0418 07:53:56.605346 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 28.771715ms to process 1 records I0418 07:53:56.605355 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 29.112693ms to process 0 records I0418 07:53:56.605365 1 tasks_processing.go:74] worker 46 stopped. I0418 07:53:56.605423 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=592822be63d824537461844d8a3c3232866474a629a12c5cef14859d9d117724 I0418 07:53:56.605434 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 29.655874ms to process 1 records I0418 07:53:56.605440 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 29.155835ms to process 0 records I0418 07:53:56.605446 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 30.84779ms to process 0 records I0418 07:53:56.605447 1 tasks_processing.go:74] worker 55 stopped. I0418 07:53:56.605455 1 tasks_processing.go:74] worker 31 stopped. I0418 07:53:56.605457 1 tasks_processing.go:74] worker 13 stopped. I0418 07:53:56.605530 1 tasks_processing.go:74] worker 34 stopped. I0418 07:53:56.605615 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=5c8489898602956907069b15cad4ad432b5abad0dc49037878ef73a6e158db0e I0418 07:53:56.605639 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=1ee34a3d209912da1298931ced39fc3b272e0eeb507648df5874ad78b64ab560 I0418 07:53:56.605654 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=4f450a096e3825e7786b3fe0e8e07f552114c19cedd5dd451d7bbc5663656cdb I0418 07:53:56.605666 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=a8f1f9027d97eda76dc968bdb0bfac3fd485df8095c5592463b88c114f4bf26e I0418 07:53:56.605682 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=3bf38aa098587bf8a690ae39a5a565a133823f535d8dc6f1858fcb8c9810c660 I0418 07:53:56.605696 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=a77a6e02f77a230705ba86e6f7b5abc0653ef1d707f3f25c9062a50feaa5eae3 I0418 07:53:56.605710 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=6124e38ef24f1d8d47c88470b0eb8c022074eafe853871d6c199237fdf39b2f2 I0418 07:53:56.605733 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=186e2f5ab8f06047961fb6bf151580afbe21e701df897bd6bb4b84cb60ce82d3 I0418 07:53:56.605748 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=c17389ae68a518340051dd6e0d63de155f625461d0b70c4edb577e285b998999 I0418 07:53:56.605762 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=ba60ed0b90f644aff290da03fef475fc8af7c7f9959783250a5f0461df94235d I0418 07:53:56.605774 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=5130be6453cf82bf20e547af2c0851f3983cec24d9a38bb433d9134f49923a53 I0418 07:53:56.605782 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 29.889311ms to process 11 records I0418 07:53:56.605788 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 31.707198ms to process 0 records I0418 07:53:56.605793 1 tasks_processing.go:74] worker 15 stopped. I0418 07:53:56.606027 1 tasks_processing.go:74] worker 1 stopped. I0418 07:53:56.606037 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 32.008951ms to process 0 records I0418 07:53:56.606918 1 tasks_processing.go:74] worker 38 stopped. I0418 07:53:56.606996 1 recorder.go:75] Recording config/proxy with fingerprint=b3d535280eeded4f34d6ed30db3fac83d7fb045b0963cb5a7413de2e6de77b4f I0418 07:53:56.607010 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 31.268608ms to process 1 records I0418 07:53:56.607091 1 tasks_processing.go:74] worker 41 stopped. I0418 07:53:56.607104 1 recorder.go:75] Recording config/image with fingerprint=47010ca1d17f2e1d052e8ae6738c550cd99700c8fe21242200c3da0f62c34d39 I0418 07:53:56.607112 1 gather.go:177] gatherer "clusterconfig" function "image" took 32.42265ms to process 1 records I0418 07:53:56.607186 1 tasks_processing.go:74] worker 39 stopped. I0418 07:53:56.607370 1 recorder.go:75] Recording config/oauth with fingerprint=677c248b8318d0ab6a11146b037d3b18c90716ab8a21efa48efc552b91c8f1a6 I0418 07:53:56.607381 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 32.327414ms to process 1 records W0418 07:53:56.607540 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0418 07:53:56.607557 1 tasks_processing.go:74] worker 52 stopped. I0418 07:53:56.607584 1 recorder.go:75] Recording config/authentication with fingerprint=89ca68455e9fcae2bf575f5c8997e98e3cd18666e8ed316b8cd011204cd5eb28 I0418 07:53:56.607591 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 32.504942ms to process 1 records I0418 07:53:56.607597 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 30.960508ms to process 0 records I0418 07:53:56.607614 1 tasks_processing.go:74] worker 50 stopped. I0418 07:53:56.607675 1 tasks_processing.go:74] worker 6 stopped. I0418 07:53:56.607801 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=f58ced925ff1f2f80bd749176881e9ec0be6667ba41c6883af3fdcd1b8aee0f9 I0418 07:53:56.607849 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=9172019bab34ab5b66d71bad4100e8ab7adf168c9e6e65e25a9706bde954353f I0418 07:53:56.607891 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=35b454017fe3d1a43425087d4304bfb062ac935dabbbbe9c1f6939f853f2cf89 I0418 07:53:56.607917 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 31.245232ms to process 3 records I0418 07:53:56.608374 1 tasks_processing.go:74] worker 10 stopped. I0418 07:53:56.610253 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=f4a2c468744b7495dbf93c946e19d2106b12e19f03f3c74f00e7805c74287a6a I0418 07:53:56.610325 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 34.23889ms to process 1 records I0418 07:53:56.610382 1 tasks_processing.go:74] worker 30 stopped. I0418 07:53:56.612014 1 recorder.go:75] Recording config/network with fingerprint=1d390ddd6e7a275b05cb8f3b0b1df72d90c20933858b1fad7c0209ef06f612cc I0418 07:53:56.612041 1 gather.go:177] gatherer "clusterconfig" function "networks" took 28.676496ms to process 1 records I0418 07:53:56.612189 1 tasks_processing.go:74] worker 22 stopped. I0418 07:53:56.612466 1 recorder.go:75] Recording config/node/ip-10-0-0-123.ec2.internal with fingerprint=af6f586e6391b7dfdad16389673a776a43e3cb214e9d84adaa7f82cac9c8bd1f I0418 07:53:56.612564 1 recorder.go:75] Recording config/node/ip-10-0-1-13.ec2.internal with fingerprint=06b9e9b34ed212b9e92874e43a62b46efc216cacf101124df224a2b5a1c130ae I0418 07:53:56.612661 1 recorder.go:75] Recording config/node/ip-10-0-2-176.ec2.internal with fingerprint=bce23f04c47b7940402d3527c38aa2d038584fe396b580106bda9fc687748eca I0418 07:53:56.612675 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 34.628282ms to process 3 records I0418 07:53:56.612756 1 tasks_processing.go:74] worker 54 stopped. I0418 07:53:56.612795 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=dd9e03cbae36f91d8b9a2ae482883ceacfefad7ae9b43500fca9a83ef3bf9518 I0418 07:53:56.612871 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=b2db69386bd79e8bf429c6c34dd96414e085748ee75743c3b571f7d4f8040415 I0418 07:53:56.613021 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=c91bd99d7efbf5296d2edc2563cb078b0966b151e8e3f6ba11ddda2488fb6031 I0418 07:53:56.613091 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=cb417de8241d5f80cb29e819fb6aa0a4987e612b46aed6d58a4c8e4fdfb5f866 I0418 07:53:56.613135 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=a220def3cd857ff1654cb3c1685d7759352af6ff82e1cbe939e33dacbfa0b226 I0418 07:53:56.613186 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=b9aede1db321ba7fba5904b2acaf72c04006472cc2568b60e89637d892a2c9c4 I0418 07:53:56.613234 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=f31d43a88c8e519e785b4387f1f35d3e14d83a20545a61efc368848daebfa479 I0418 07:53:56.613309 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=e7781e4fdd444a0894371354e6bc14a116680b508fd1bed237150ec64f8d3aac I0418 07:53:56.613355 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=067a1bbe0d89b7f2f403a8b9553357dd765e4369fa6e969fa4c826faec084bf8 I0418 07:53:56.613422 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=6966cab0352673794a2428e8d15de9cf80c147a53dfb248cc8e2f73aa8a51d23 I0418 07:53:56.613470 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=6378680d0bce9cd39aa749ad2422b4fa47bb8b99b1096a839c270cbaa138f1ea I0418 07:53:56.613518 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=f00c36c8050de0987d98b885f44d0ca32c85e6fa5306d98048e68531604cf5ec I0418 07:53:56.613565 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=bf98a1664081b1e5896b50784c02bc0dac782e5ac1bb2bc54bb16c4afdbc7a7d I0418 07:53:56.613576 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 35.994918ms to process 13 records I0418 07:53:56.613605 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0418 07:53:56.613621 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 39.655393ms to process 1 records I0418 07:53:56.613633 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 37.903724ms to process 0 records I0418 07:53:56.613651 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0418 07:53:56.613662 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 38.618223ms to process 1 records I0418 07:53:56.613681 1 tasks_processing.go:74] worker 59 stopped. I0418 07:53:56.613676 1 tasks_processing.go:74] worker 19 stopped. I0418 07:53:56.613680 1 tasks_processing.go:74] worker 61 stopped. I0418 07:53:56.615591 1 tasks_processing.go:74] worker 5 stopped. I0418 07:53:56.615606 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 41.45539ms to process 0 records I0418 07:53:56.619507 1 tasks_processing.go:74] worker 14 stopped. I0418 07:53:56.619636 1 recorder.go:75] Recording config/olm_operators with fingerprint=64ae30f349de9d3fbeb4d37dd7eb10dbd1b1df63f5be1dc1b483d9a7241c8f1d I0418 07:53:56.619656 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 45.493226ms to process 1 records I0418 07:53:56.620151 1 tasks_processing.go:74] worker 43 stopped. I0418 07:53:56.620394 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=a113e0fd295c4e6247d55ce446643a2733c4987f95bb95d7f44742d07ed9cf15 I0418 07:53:56.620532 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=b22fa82b9cb93252e3d9c85bf992962c852118ea697fdb4d471926549e24921e I0418 07:53:56.620545 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 45.616588ms to process 2 records I0418 07:53:56.621379 1 tasks_processing.go:74] worker 45 stopped. I0418 07:53:56.621687 1 recorder.go:75] Recording config/version with fingerprint=7e993b17c6cac893cf777343e2b87d8a7878e67d91266c404762b2938af157f2 I0418 07:53:56.621705 1 recorder.go:75] Recording config/id with fingerprint=67e25a76ab67519efe69d10c22f64d838b8b7cf9700588de6912bd518aa41580 I0418 07:53:56.621713 1 gather.go:177] gatherer "clusterconfig" function "version" took 47.157048ms to process 2 records I0418 07:53:56.624993 1 tasks_processing.go:74] worker 49 stopped. I0418 07:53:56.625697 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=0d20fae3e00ae68664c7a737401292909e76aae373c59f73c81d50384e8f4c2c I0418 07:53:56.626133 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=5ac88cfc4a5619bb674a0f9452287cfab08aa2e780bf1221f66b3bdd7965e26a I0418 07:53:56.626150 1 gather.go:177] gatherer "clusterconfig" function "crds" took 50.550614ms to process 2 records I0418 07:53:56.626166 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 49.939764ms to process 0 records I0418 07:53:56.626176 1 tasks_processing.go:74] worker 56 stopped. I0418 07:53:56.631737 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0418 07:53:56.633935 1 tasks_processing.go:74] worker 16 stopped. I0418 07:53:56.633954 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 59.762817ms to process 0 records I0418 07:53:56.634479 1 tasks_processing.go:74] worker 51 stopped. E0418 07:53:56.634493 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0418 07:53:56.634499 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0418 07:53:56.634503 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0418 07:53:56.634512 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0418 07:53:56.634538 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0418 07:53:56.634545 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0418 07:53:56.634549 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0418 07:53:56.634572 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0418 07:53:56.634611 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0418 07:53:56.634619 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0418 07:53:56.634624 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 59.882382ms to process 7 records I0418 07:53:56.634706 1 tasks_processing.go:74] worker 63 stopped. I0418 07:53:56.635862 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-ct6sv with fingerprint=96dbc5007720819db5ada72980d39c52191cffc9d275cc613f3814b472613624 I0418 07:53:56.635976 1 recorder.go:75] Recording config/pod/openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-1-13.ec2.internal with fingerprint=73865dd15c06cdb8079abfb651550d7c6dc6b65ec080ead9cceb273d1d93ef39 I0418 07:53:56.636048 1 recorder.go:75] Recording config/running_containers with fingerprint=cc77af2dc86d7d15da96c3a1ace33db1389ff5aebc2661e6c94ab931680bb1c3 I0418 07:53:56.636057 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 61.249743ms to process 3 records I0418 07:53:56.639928 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0418 07:53:56.639940 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0418 07:53:56.639951 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0418 07:53:56.646127 1 base_controller.go:82] Caches are synced for ConfigController I0418 07:53:56.646138 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0418 07:53:56.651436 1 configmapobserver.go:84] configmaps "insights-config" not found I0418 07:53:56.679215 1 base_controller.go:82] Caches are synced for LoggingSyncer I0418 07:53:56.679228 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0418 07:53:56.721067 1 cluster_transfer.go:102] no available accepted cluster transfer I0418 07:53:56.721082 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0418 07:53:56.725824 1 tasks_processing.go:74] worker 29 stopped. E0418 07:53:56.725836 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0418 07:53:56.725842 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2poebv4lqpsihskk0anl4qje6l4tp2si-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2poebv4lqpsihskk0anl4qje6l4tp2si-primary-cert-bundle-secret" not found I0418 07:53:56.725918 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=7aadefeac81cda141076fd12083417e1a1edeba688c187867df434aab00e2e64 I0418 07:53:56.725932 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 151.225049ms to process 1 records I0418 07:53:56.799282 1 sca.go:266] etc-pki-entitlement secret successfully created I0418 07:53:56.799296 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated W0418 07:53:57.607059 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0418 07:53:57.654231 1 tasks_processing.go:74] worker 62 stopped. I0418 07:53:57.654303 1 recorder.go:75] Recording config/alerts with fingerprint=8115c87ef9d3f77b79a1d5b75091e34e2d8a53f6312fa8dfaa0bc88930b9f507 I0418 07:53:57.654323 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.079585478s to process 1 records I0418 07:53:57.654488 1 tasks_processing.go:74] worker 12 stopped. I0418 07:53:57.654504 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0418 07:53:57.654513 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.078528576s to process 1 records W0418 07:53:58.607520 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0418 07:53:59.038477 1 tasks_processing.go:74] worker 7 stopped. I0418 07:53:59.038589 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=cf47089bb6a981daf748c35420b28191e444c4c7843c264c61c7c63bd5ef05a1 I0418 07:53:59.038635 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0418 07:53:59.038662 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=04684f2adc5ee868a7b4a775e04cecf46e7681ed9c4679ae5098db25e8e08f7b I0418 07:53:59.038673 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0418 07:53:59.038694 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=00da0ac64895e44ab8bf9edb0f768db6e31f0c8c5d3f835c7b3e5d2976660593 I0418 07:53:59.038710 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0418 07:53:59.038730 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=44b890b8c1f6283eea03b0b8872af7a9691140ca1da328f8bc1d685a6039b258 I0418 07:53:59.038761 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=25956561ace4c75736d4e13ad3202daa85aa443d35565550a524673e48fa3fc9 I0418 07:53:59.038784 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=98da3195f7970d23e28d5db6fc30a746e69399600842c2bcf20782589c9d77a1 I0418 07:53:59.038814 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=aca6087bfbe924b8e962ed30c6177bbaf15cf731a6a29b3dfdb8242ef51a647b I0418 07:53:59.039098 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=60c80dbbee5ddeffa511dd1e4f5f74205f54ee4ad470f3cd9aa0172d888f5de8 I0418 07:53:59.039121 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=a12af19e1c5cdde4fbb8b69abb1ace9d8a6f465a0b4becc6ef7d37dcbb43f899 I0418 07:53:59.039130 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0418 07:53:59.039146 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=bea7fca1956b20860fafddd1e7921bfb14ba48416df0f657dead3541cb327413 I0418 07:53:59.039157 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0418 07:53:59.039173 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=00827142dcb95dcdcbf7470b601ccc1e1e8432ae59d923438b470043fcab1881 I0418 07:53:59.039180 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0418 07:53:59.039199 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=2c6c1246b29477e6cc63e9e0d458aaf4c84288f15109c5fb2b05c9fe66cb07af I0418 07:53:59.039209 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0418 07:53:59.039231 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=fd87a537131eb13efccaaebde65eaf5b2b8873a4c479dae0cfe7befb85c2c2b6 I0418 07:53:59.039333 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=910309a9653584655773fd0f561952038eb8b30f2626fdb53efcecad26548376 I0418 07:53:59.039354 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0418 07:53:59.039363 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0418 07:53:59.039385 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0418 07:53:59.039407 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=8351732f225a514a9224ae3ca4d46edb050ffd6f7925b89d1b4e14bf174cc812 I0418 07:53:59.039430 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=88ba19eb3f045dee857f50d762a517e894a79358d3f4072492990521832dde0c I0418 07:53:59.039439 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0418 07:53:59.039454 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=51f47bbd5fd32dbb3be902de5b9e29bec186924ba95f44b93a8d173285512337 I0418 07:53:59.039464 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0418 07:53:59.039481 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=96c632d5696027fafc722032b570af4c6b829dc328ee5965e31432330fa03244 I0418 07:53:59.039490 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0418 07:53:59.039502 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=3b3b5b4352015032981d8e0c7b8b158e856694ca92746169961f98033a40a66c I0418 07:53:59.039515 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=b99a16e2e5ba37fe43c1227ff9f748f9a34104762d4a6e84c1d55f10a2e998c7 I0418 07:53:59.039531 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=c930341d5797fe2e7c76629d234912fdc5836ed0512fdd9bc6ca4e1ec194c63d I0418 07:53:59.039553 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=d0dab1ac965bd1f6c193ffbd9f2017e90e89ed8882ba5b3a91d07e1d15f61a8e I0418 07:53:59.039563 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0418 07:53:59.039600 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=638d1f7a18c4ed0a0ffd42554137a87ed9d091c00197979593f033a8e91ee8dc I0418 07:53:59.039619 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0418 07:53:59.039627 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0418 07:53:59.039634 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.464392429s to process 39 records W0418 07:53:59.608119 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0418 07:54:00.607026 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0418 07:54:01.607643 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0418 07:54:01.607676 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0418 07:54:01.607697 1 tasks_processing.go:74] worker 35 stopped. E0418 07:54:01.607710 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0418 07:54:01.607723 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0418 07:54:01.607737 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.032795243s to process 1 records I0418 07:54:09.418235 1 tasks_processing.go:74] worker 27 stopped. I0418 07:54:09.418287 1 recorder.go:75] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0418 07:54:09.418299 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.843739919s to process 1 records I0418 07:54:10.181736 1 tasks_processing.go:74] worker 42 stopped. I0418 07:54:10.182029 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=4aff5860ec073ddfdf1bec96303c96ef7753a0d476f8315de0a2062bde3a2fd5 I0418 07:54:10.182045 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.607121968s to process 1 records E0418 07:54:10.182105 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.609s with: function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"machines\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0418 07:54:10.183214 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "tsdb_status" failed with an error, function "metrics" failed with an error, function "machines" failed with an error, function "machine_healthchecks" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0418 07:54:10.183231 1 periodic.go:209] Running workloads gatherer I0418 07:54:10.183252 1 tasks_processing.go:45] number of workers: 2 I0418 07:54:10.183270 1 tasks_processing.go:69] worker 1 listening for tasks. I0418 07:54:10.183278 1 tasks_processing.go:71] worker 1 working on workload_info task. I0418 07:54:10.183278 1 tasks_processing.go:69] worker 0 listening for tasks. I0418 07:54:10.183297 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0418 07:54:10.189066 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-13.ec2.internal... I0418 07:54:10.189086 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-123.ec2.internal... I0418 07:54:10.189072 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-176.ec2.internal... I0418 07:54:10.208036 1 tasks_processing.go:74] worker 0 stopped. I0418 07:54:10.208051 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 24.718594ms to process 0 records I0418 07:54:15.481819 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 5s I0418 07:54:15.505297 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0418 07:54:15.513809 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (10ms) I0418 07:54:15.522965 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (9ms) I0418 07:54:15.531645 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (9ms) I0418 07:54:15.540172 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (8ms) I0418 07:54:15.549211 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (9ms) I0418 07:54:15.557770 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (9ms) I0418 07:54:15.566314 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (9ms) I0418 07:54:15.575118 1 gather_workloads_info.go:398] Found image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (9ms) I0418 07:54:15.583779 1 gather_workloads_info.go:387] No image sha256:1477f3287cc75352305274f248f781c1b7c2a8696f4f7bc450c45a4a5f5b10ea (8ms) I0418 07:54:15.600993 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (17ms) I0418 07:54:15.612636 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (12ms) I0418 07:54:15.714152 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (102ms) I0418 07:54:15.815340 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (101ms) I0418 07:54:15.914010 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (99ms) I0418 07:54:16.014015 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (100ms) I0418 07:54:16.114825 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (101ms) I0418 07:54:16.214755 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0418 07:54:16.314869 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0418 07:54:16.415980 1 gather_workloads_info.go:387] No image sha256:7bd3e88d6d3f878a89203df590b684c5a77bf08df4717451d3197ab74683a80e (101ms) I0418 07:54:16.513617 1 gather_workloads_info.go:387] No image sha256:ce98d5d844bfc2ba8de1893866ad38166c95157d54abd8192b181e819bc50bb5 (98ms) I0418 07:54:16.613891 1 gather_workloads_info.go:387] No image sha256:90bdf9e678fe18e35401e34e1cb0912d822e9cb32b9e41cc545870540e9a87f2 (100ms) I0418 07:54:16.715084 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (101ms) I0418 07:54:16.813756 1 gather_workloads_info.go:387] No image sha256:6e5df2aab9ce759e8f957341a11c087c233307e3c8f8760d21270efc19ccd3ce (99ms) I0418 07:54:16.916279 1 gather_workloads_info.go:387] No image sha256:101993f6e65d6fda8b90c583b718b720809182961f69bc777295f47e6df4d7ce (103ms) I0418 07:54:17.017814 1 gather_workloads_info.go:387] No image sha256:1e65f9f5490fbc930d2dce96f03ddcb28990b839d7dc011ef5de5bec6c5095bf (102ms) I0418 07:54:17.115027 1 gather_workloads_info.go:387] No image sha256:74713675d580a47a56b628469c4c05c41e114e0b935960d718ff0cf7e72f613d (97ms) I0418 07:54:17.214448 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (99ms) I0418 07:54:17.315159 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (101ms) I0418 07:54:17.414471 1 gather_workloads_info.go:398] Found image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (99ms) I0418 07:54:17.514752 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (100ms) I0418 07:54:17.614557 1 gather_workloads_info.go:387] No image sha256:513b1166e376feca472fdd5174f6abf0abf02925d6a21b7b4d89a2ff2fc67bed (100ms) I0418 07:54:17.715180 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (101ms) I0418 07:54:17.814271 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (99ms) I0418 07:54:17.913850 1 gather_workloads_info.go:387] No image sha256:0af73ad7a47c9ff53269ae2bc5163bf9e389085220c34e4bd42c92c4e7a85f6d (100ms) I0418 07:54:18.014968 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (101ms) I0418 07:54:18.115872 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (101ms) I0418 07:54:18.215460 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (100ms) I0418 07:54:18.314876 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (99ms) I0418 07:54:18.414206 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (99ms) I0418 07:54:18.514809 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (101ms) I0418 07:54:18.614963 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (100ms) I0418 07:54:18.714378 1 gather_workloads_info.go:387] No image sha256:0092c6ff139f8698c91349d507eb3182bc5d6c4ae040f672e347ff504accdea2 (99ms) I0418 07:54:18.814555 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (100ms) I0418 07:54:18.915017 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (100ms) I0418 07:54:19.014474 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (99ms) I0418 07:54:19.114261 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0418 07:54:19.214623 1 gather_workloads_info.go:387] No image sha256:b99613494e9d6a4a8207323872d63cc02b830fc80a8bf972e62c64381aa4cd17 (100ms) I0418 07:54:19.314805 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0418 07:54:19.314835 1 tasks_processing.go:74] worker 1 stopped. I0418 07:54:19.315151 1 recorder.go:75] Recording config/workload_info with fingerprint=50b606275312c16ee9620e3f35bb66cc0ce84b72fc6e5f29bf7890f9a6cee79a I0418 07:54:19.315166 1 gather.go:177] gatherer "workloads" function "workload_info" took 9.13154982s to process 1 records I0418 07:54:19.315179 1 periodic.go:256] Periodic gather workloads completed in 9.131s I0418 07:54:19.315190 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0418 07:54:19.315194 1 periodic.go:209] Running conditional gatherer I0418 07:54:19.320961 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0418 07:54:19.493109 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0418 07:54:20.577176 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0418 07:54:20.583563 1 conditional_gatherer.go:392] cluster version is '4.20.8' I0418 07:54:20.583590 1 tasks_processing.go:45] number of workers: 3 I0418 07:54:20.583602 1 tasks_processing.go:69] worker 2 listening for tasks. I0418 07:54:20.583606 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0418 07:54:20.583606 1 tasks_processing.go:69] worker 0 listening for tasks. I0418 07:54:20.583620 1 tasks_processing.go:69] worker 1 listening for tasks. I0418 07:54:20.583627 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0418 07:54:20.583629 1 tasks_processing.go:74] worker 1 stopped. I0418 07:54:20.583648 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0418 07:54:20.583688 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0418 07:54:20.583700 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.643µs to process 1 records I0418 07:54:20.583732 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0418 07:54:20.583740 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.517µs to process 1 records I0418 07:54:20.583745 1 tasks_processing.go:74] worker 0 stopped. I0418 07:54:20.583938 1 tasks_processing.go:74] worker 2 stopped. I0418 07:54:20.583951 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 266.102µs to process 0 records I0418 07:54:20.583977 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0418 07:54:20.584000 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0418 07:54:20.607614 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=1c8a104e96060df117feb47da3afc0af8a979f6c5b78071b2483350427b1b1a0 I0418 07:54:20.607791 1 diskrecorder.go:70] Writing 112 records to /var/lib/insights-operator/insights-2026-04-18-075420.tar.gz I0418 07:54:20.614690 1 diskrecorder.go:51] Wrote 112 records to disk in 6ms I0418 07:54:20.614718 1 periodic.go:278] Gathering cluster info every 2h0m0s I0418 07:54:20.614734 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0418 07:55:26.572625 1 diskrecorder.go:223] Found files to send: insights-2026-04-18-075420.tar.gz I0418 07:55:26.572653 1 insightsuploader.go:150] Checking archives to upload periodically every 16m11.331639842s I0418 07:55:26.572663 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0418 07:55:26.585600 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0418 07:55:26.818795 1 requests.go:87] Successfully reported id=2026-04-18T07:55:26Z x-rh-insights-request-id=44ddfce5aa8c4e87a73477b5dc419c98, wrote=50248 I0418 07:55:26.818928 1 insightsuploader.go:187] Uploaded report successfully in 246.25742ms I0418 07:55:26.818952 1 controller.go:128] Initializing last reported time to 2026-04-18T07:55:26Z I0418 07:55:26.819024 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0418 07:55:26.819035 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0418 07:55:26.819043 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0418 07:55:26.823750 1 controller.go:512] The operator is healthy I0418 07:55:56.584832 1 controller.go:512] The operator is healthy I0418 07:56:28.449089 1 insightsreport.go:137] Pulling report from smart-proxy I0418 07:56:28.449118 1 insightsreport.go:149] Retrieving report I0418 07:56:28.455806 1 requests.go:110] Retrieving report for cluster: 552ea46f-895a-41a4-b70f-0a2a6cb14274 I0418 07:56:28.455817 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/552ea46f-895a-41a4-b70f-0a2a6cb14274/reports I0418 07:56:28.458558 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/552ea46f-895a-41a4-b70f-0a2a6cb14274/reports I0418 07:56:28.644486 1 insightsreport.go:184] Report retrieved I0418 07:56:28.668556 1 insightsreport.go:239] Report retrieved correctly I0418 07:57:56.586535 1 controller.go:512] The operator is healthy I0418 07:58:56.566731 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0418 07:58:56.572018 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0418 07:58:56.572035 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0418 07:58:56.576031 1 secretconfigobserver.go:119] support secret does not exist I0418 07:59:56.584691 1 controller.go:512] The operator is healthy I0418 08:01:56.585219 1 controller.go:512] The operator is healthy I0418 08:03:56.568282 1 configmapobserver.go:84] configmaps "insights-config" not found I0418 08:03:56.576355 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0418 08:03:56.589368 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0418 08:03:56.589396 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0418 08:03:56.589642 1 controller.go:512] The operator is healthy I0418 08:03:56.598330 1 secretconfigobserver.go:119] support secret does not exist I0418 08:05:56.585397 1 controller.go:512] The operator is healthy I0418 08:07:56.585593 1 controller.go:512] The operator is healthy I0418 08:08:56.598875 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0418 08:08:56.604735 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0418 08:08:56.604754 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0418 08:08:56.608931 1 secretconfigobserver.go:119] support secret does not exist I0418 08:09:56.584699 1 controller.go:512] The operator is healthy I0418 08:11:43.151594 1 insightsuploader.go:146] Nothing to report since 2026-04-18T07:55:26Z I0418 08:11:56.584719 1 controller.go:512] The operator is healthy I0418 08:13:56.561129 1 configmapobserver.go:84] configmaps "insights-config" not found I0418 08:13:56.564909 1 configmapobserver.go:84] configmaps "insights-config" not found I0418 08:13:56.584227 1 controller.go:512] The operator is healthy I0418 08:13:56.608991 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0418 08:13:56.615266 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0418 08:13:56.615284 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0418 08:13:56.618603 1 secretconfigobserver.go:119] support secret does not exist I0418 08:15:56.589251 1 controller.go:512] The operator is healthy I0418 08:17:56.589713 1 controller.go:512] The operator is healthy I0418 08:18:56.619190 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0418 08:18:56.628635 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0418 08:18:56.628658 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0418 08:18:56.635297 1 secretconfigobserver.go:119] support secret does not exist I0418 08:19:56.588944 1 controller.go:512] The operator is healthy I0418 08:21:56.590102 1 controller.go:512] The operator is healthy I0418 08:23:56.563989 1 configmapobserver.go:84] configmaps "insights-config" not found I0418 08:23:56.570172 1 configmapobserver.go:84] configmaps "insights-config" not found I0418 08:23:56.587056 1 controller.go:512] The operator is healthy I0418 08:23:56.636051 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0418 08:23:56.642635 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0418 08:23:56.642651 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0418 08:23:56.648598 1 secretconfigobserver.go:119] support secret does not exist I0418 08:25:56.587279 1 controller.go:512] The operator is healthy I0418 08:27:46.257067 1 insightsuploader.go:146] Nothing to report since 2026-04-18T07:55:26Z I0418 08:27:56.586931 1 controller.go:512] The operator is healthy I0418 08:28:56.649652 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0418 08:28:56.657308 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0418 08:28:56.657327 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0418 08:28:56.663517 1 secretconfigobserver.go:119] support secret does not exist I0418 08:29:56.587001 1 controller.go:512] The operator is healthy