I0430 23:28:29.317155 1 cmd.go:253] Using service-serving-cert provided certificates I0430 23:28:29.317615 1 observer_polling.go:159] Starting file observer I0430 23:28:29.772340 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0430 23:28:29.772537 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0430 23:28:29.773096 1 secure_serving.go:57] Forcing use of http/1.1 only W0430 23:28:29.773118 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0430 23:28:29.773124 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0430 23:28:29.773128 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0430 23:28:29.773131 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0430 23:28:29.773133 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0430 23:28:29.773136 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0430 23:28:29.773159 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0430 23:28:29.777179 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0430 23:28:29.777209 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"8246df7d-a23e-489a-ab0b-a11b08bd4a2d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0430 23:28:29.777863 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0430 23:28:29.777879 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0430 23:28:29.777885 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0430 23:28:29.777900 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0430 23:28:29.777903 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0430 23:28:29.777917 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0430 23:28:29.778255 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0430 23:28:29.778722 1 secure_serving.go:213] Serving securely on [::]:8443 I0430 23:28:29.778755 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0430 23:28:29.781856 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0430 23:28:29.781880 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0430 23:28:29.781998 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0430 23:28:29.786974 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0430 23:28:29.787247 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0430 23:28:29.794127 1 secretconfigobserver.go:119] support secret does not exist I0430 23:28:29.799760 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0430 23:28:29.803902 1 secretconfigobserver.go:119] support secret does not exist I0430 23:28:29.805581 1 recorder.go:176] Pruning old reports every 4h52m21s, max age is 288h0m0s I0430 23:28:29.810317 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0430 23:28:29.810319 1 periodic.go:216] Running clusterconfig gatherer I0430 23:28:29.810332 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0430 23:28:29.810350 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0430 23:28:29.810354 1 insightsreport.go:296] Starting report retriever I0430 23:28:29.810356 1 tasks_processing.go:45] number of workers: 64 I0430 23:28:29.810359 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0430 23:28:29.810367 1 tasks_processing.go:69] worker 1 listening for tasks. I0430 23:28:29.810373 1 tasks_processing.go:69] worker 0 listening for tasks. I0430 23:28:29.810381 1 tasks_processing.go:69] worker 12 listening for tasks. I0430 23:28:29.810387 1 tasks_processing.go:69] worker 9 listening for tasks. I0430 23:28:29.810387 1 tasks_processing.go:69] worker 8 listening for tasks. I0430 23:28:29.810398 1 tasks_processing.go:69] worker 10 listening for tasks. I0430 23:28:29.810403 1 tasks_processing.go:69] worker 2 listening for tasks. I0430 23:28:29.810406 1 tasks_processing.go:69] worker 13 listening for tasks. I0430 23:28:29.810409 1 tasks_processing.go:69] worker 6 listening for tasks. I0430 23:28:29.810417 1 tasks_processing.go:69] worker 7 listening for tasks. I0430 23:28:29.810412 1 tasks_processing.go:69] worker 5 listening for tasks. I0430 23:28:29.810406 1 tasks_processing.go:69] worker 14 listening for tasks. I0430 23:28:29.810427 1 tasks_processing.go:69] worker 4 listening for tasks. I0430 23:28:29.810423 1 tasks_processing.go:69] worker 11 listening for tasks. I0430 23:28:29.810427 1 tasks_processing.go:69] worker 3 listening for tasks. I0430 23:28:29.810447 1 tasks_processing.go:69] worker 15 listening for tasks. I0430 23:28:29.810480 1 tasks_processing.go:71] worker 15 working on validating_webhook_configurations task. I0430 23:28:29.810491 1 tasks_processing.go:69] worker 47 listening for tasks. I0430 23:28:29.810499 1 tasks_processing.go:71] worker 47 working on image_pruners task. I0430 23:28:29.810503 1 tasks_processing.go:71] worker 1 working on feature_gates task. I0430 23:28:29.810507 1 tasks_processing.go:71] worker 6 working on silenced_alerts task. I0430 23:28:29.810509 1 tasks_processing.go:69] worker 63 listening for tasks. I0430 23:28:29.810516 1 tasks_processing.go:69] worker 31 listening for tasks. I0430 23:28:29.810522 1 tasks_processing.go:69] worker 16 listening for tasks. I0430 23:28:29.810535 1 tasks_processing.go:69] worker 23 listening for tasks. I0430 23:28:29.810530 1 tasks_processing.go:69] worker 48 listening for tasks. I0430 23:28:29.810534 1 tasks_processing.go:69] worker 49 listening for tasks. I0430 23:28:29.810547 1 tasks_processing.go:69] worker 51 listening for tasks. I0430 23:28:29.810551 1 tasks_processing.go:69] worker 19 listening for tasks. I0430 23:28:29.810539 1 tasks_processing.go:69] worker 50 listening for tasks. I0430 23:28:29.810557 1 tasks_processing.go:69] worker 20 listening for tasks. I0430 23:28:29.810561 1 tasks_processing.go:69] worker 26 listening for tasks. I0430 23:28:29.810564 1 tasks_processing.go:69] worker 53 listening for tasks. I0430 23:28:29.810567 1 tasks_processing.go:69] worker 18 listening for tasks. I0430 23:28:29.810557 1 tasks_processing.go:69] worker 52 listening for tasks. I0430 23:28:29.810573 1 tasks_processing.go:69] worker 27 listening for tasks. I0430 23:28:29.810556 1 tasks_processing.go:69] worker 25 listening for tasks. I0430 23:28:29.810581 1 tasks_processing.go:69] worker 56 listening for tasks. I0430 23:28:29.810580 1 tasks_processing.go:69] worker 22 listening for tasks. I0430 23:28:29.810579 1 tasks_processing.go:69] worker 17 listening for tasks. I0430 23:28:29.810589 1 tasks_processing.go:69] worker 55 listening for tasks. I0430 23:28:29.810571 1 tasks_processing.go:69] worker 21 listening for tasks. I0430 23:28:29.810594 1 tasks_processing.go:69] worker 62 listening for tasks. I0430 23:28:29.810597 1 tasks_processing.go:71] worker 12 working on nodenetworkconfigurationpolicies task. I0430 23:28:29.810595 1 tasks_processing.go:69] worker 58 listening for tasks. I0430 23:28:29.810544 1 tasks_processing.go:69] worker 24 listening for tasks. I0430 23:28:29.810596 1 tasks_processing.go:71] worker 0 working on machine_autoscalers task. I0430 23:28:29.810604 1 tasks_processing.go:69] worker 30 listening for tasks. I0430 23:28:29.810610 1 tasks_processing.go:69] worker 57 listening for tasks. I0430 23:28:29.810607 1 tasks_processing.go:71] worker 8 working on container_images task. I0430 23:28:29.810609 1 tasks_processing.go:69] worker 61 listening for tasks. I0430 23:28:29.810619 1 tasks_processing.go:69] worker 35 listening for tasks. I0430 23:28:29.810618 1 tasks_processing.go:69] worker 32 listening for tasks. I0430 23:28:29.810617 1 tasks_processing.go:69] worker 43 listening for tasks. I0430 23:28:29.810581 1 tasks_processing.go:69] worker 59 listening for tasks. I0430 23:28:29.810587 1 tasks_processing.go:69] worker 60 listening for tasks. I0430 23:28:29.810587 1 tasks_processing.go:69] worker 28 listening for tasks. I0430 23:28:29.810634 1 tasks_processing.go:69] worker 41 listening for tasks. I0430 23:28:29.810626 1 tasks_processing.go:71] worker 2 working on ingress task. I0430 23:28:29.810635 1 tasks_processing.go:69] worker 34 listening for tasks. I0430 23:28:29.810643 1 tasks_processing.go:69] worker 42 listening for tasks. I0430 23:28:29.810591 1 tasks_processing.go:71] worker 9 working on sap_datahubs task. I0430 23:28:29.810646 1 tasks_processing.go:69] worker 44 listening for tasks. I0430 23:28:29.810656 1 tasks_processing.go:71] worker 11 working on qemu_kubevirt_launcher_logs task. I0430 23:28:29.810658 1 tasks_processing.go:71] worker 42 working on ingress_certificates task. I0430 23:28:29.810666 1 tasks_processing.go:71] worker 7 working on storage_cluster task. I0430 23:28:29.810710 1 tasks_processing.go:69] worker 37 listening for tasks. I0430 23:28:29.810764 1 tasks_processing.go:71] worker 63 working on node_features task. I0430 23:28:29.810794 1 tasks_processing.go:71] worker 16 working on openstack_dataplanenodesets task. I0430 23:28:29.810807 1 tasks_processing.go:71] worker 57 working on storage_classes task. I0430 23:28:29.810816 1 tasks_processing.go:71] worker 51 working on schedulers task. I0430 23:28:29.810878 1 tasks_processing.go:71] worker 59 working on certificate_signing_requests task. I0430 23:28:29.810987 1 tasks_processing.go:71] worker 28 working on config_maps task. I0430 23:28:29.810733 1 tasks_processing.go:71] worker 4 working on machine_configs task. I0430 23:28:29.811110 1 tasks_processing.go:71] worker 60 working on infrastructures task. I0430 23:28:29.811174 1 tasks_processing.go:71] worker 41 working on image task. I0430 23:28:29.810727 1 tasks_processing.go:69] worker 45 listening for tasks. I0430 23:28:29.811313 1 tasks_processing.go:71] worker 45 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0430 23:28:29.811324 1 tasks_processing.go:71] worker 34 working on nodes task. I0430 23:28:29.810761 1 tasks_processing.go:71] worker 3 working on openstack_controlplanes task. I0430 23:28:29.810573 1 tasks_processing.go:69] worker 54 listening for tasks. I0430 23:28:29.810596 1 tasks_processing.go:69] worker 29 listening for tasks. I0430 23:28:29.810602 1 tasks_processing.go:71] worker 10 working on proxies task. I0430 23:28:29.811520 1 tasks_processing.go:71] worker 61 working on install_plans task. I0430 23:28:29.810769 1 tasks_processing.go:71] worker 37 working on networks task. I0430 23:28:29.810504 1 tasks_processing.go:71] worker 14 working on image_registries task. I0430 23:28:29.811657 1 tasks_processing.go:71] worker 32 working on openstack_version task. I0430 23:28:29.811691 1 tasks_processing.go:71] worker 24 working on mutating_webhook_configurations task. I0430 23:28:29.811159 1 tasks_processing.go:71] worker 56 working on service_accounts task. I0430 23:28:29.810624 1 tasks_processing.go:69] worker 40 listening for tasks. I0430 23:28:29.810630 1 tasks_processing.go:69] worker 36 listening for tasks. I0430 23:28:29.810499 1 tasks_processing.go:71] worker 13 working on cluster_apiserver task. I0430 23:28:29.810611 1 tasks_processing.go:69] worker 39 listening for tasks. I0430 23:28:29.810748 1 tasks_processing.go:71] worker 19 working on authentication task. I0430 23:28:29.812247 1 tasks_processing.go:71] worker 30 working on aggregated_monitoring_cr_names task. I0430 23:28:29.812338 1 tasks_processing.go:71] worker 39 working on nodenetworkstates task. I0430 23:28:29.812387 1 tasks_processing.go:71] worker 43 working on node_logs task. I0430 23:28:29.812425 1 tasks_processing.go:71] worker 26 working on pdbs task. I0430 23:28:29.812476 1 tasks_processing.go:71] worker 25 working on lokistack task. I0430 23:28:29.811666 1 tasks_processing.go:71] worker 22 working on openstack_dataplanedeployments task. I0430 23:28:29.810760 1 tasks_processing.go:69] worker 46 listening for tasks. I0430 23:28:29.812642 1 tasks_processing.go:71] worker 46 working on cost_management_metrics_configs task. I0430 23:28:29.810627 1 tasks_processing.go:69] worker 33 listening for tasks. I0430 23:28:29.810785 1 tasks_processing.go:71] worker 23 working on sap_pods task. I0430 23:28:29.810780 1 tasks_processing.go:71] worker 5 working on metrics task. I0430 23:28:29.810789 1 tasks_processing.go:71] worker 31 working on machines task. I0430 23:28:29.810790 1 tasks_processing.go:71] worker 49 working on overlapping_namespace_uids task. I0430 23:28:29.810794 1 tasks_processing.go:71] worker 48 working on operators task. I0430 23:28:29.811672 1 tasks_processing.go:71] worker 17 working on clusterroles task. I0430 23:28:29.811679 1 tasks_processing.go:71] worker 55 working on pod_network_connectivity_checks task. I0430 23:28:29.811682 1 tasks_processing.go:71] worker 21 working on container_runtime_configs task. I0430 23:28:29.811685 1 tasks_processing.go:71] worker 62 working on active_alerts task. I0430 23:28:29.811688 1 tasks_processing.go:71] worker 58 working on operators_pods_and_events task. I0430 23:28:29.812370 1 tasks_processing.go:71] worker 29 working on openshift_machine_api_events task. I0430 23:28:29.812444 1 tasks_processing.go:71] worker 18 working on olm_operators task. I0430 23:28:29.812360 1 tasks_processing.go:71] worker 35 working on machine_sets task. I0430 23:28:29.812375 1 tasks_processing.go:71] worker 54 working on tsdb_status task. I0430 23:28:29.812420 1 tasks_processing.go:71] worker 20 working on machine_config_pools task. I0430 23:28:29.812412 1 tasks_processing.go:71] worker 50 working on monitoring_persistent_volumes task. I0430 23:28:29.812406 1 tasks_processing.go:71] worker 53 working on oauths task. I0430 23:28:29.814511 1 tasks_processing.go:71] worker 33 working on version task. I0430 23:28:29.812446 1 tasks_processing.go:71] worker 36 working on sap_config task. I0430 23:28:29.812421 1 tasks_processing.go:71] worker 40 working on dvo_metrics task. I0430 23:28:29.812459 1 tasks_processing.go:71] worker 27 working on jaegers task. I0430 23:28:29.812437 1 tasks_processing.go:71] worker 52 working on crds task. I0430 23:28:29.810778 1 tasks_processing.go:69] worker 38 listening for tasks. I0430 23:28:29.810660 1 tasks_processing.go:71] worker 44 working on openshift_logging task. I0430 23:28:29.815090 1 tasks_processing.go:71] worker 38 working on machine_healthchecks task. I0430 23:28:29.817988 1 tasks_processing.go:71] worker 12 working on ceph_cluster task. I0430 23:28:29.818000 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 7.37715ms to process 0 records I0430 23:28:29.818191 1 tasks_processing.go:71] worker 1 working on support_secret task. I0430 23:28:29.818338 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0430 23:28:29.818354 1 controller.go:368] The initial operator extension status is healthy I0430 23:28:29.818362 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0430 23:28:29.818362 1 recorder.go:75] Recording config/featuregate with fingerprint=9c2ad3a2d399f205c80ab732fb3904e0bbf7306d847a21ef1e5b82dbf6f9386b I0430 23:28:29.818367 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0430 23:28:29.818371 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0430 23:28:29.818377 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 7.672484ms to process 1 records I0430 23:28:29.818386 1 controller.go:531] The operator is still being initialized I0430 23:28:29.818391 1 controller.go:554] The operator is healthy I0430 23:28:29.818464 1 tasks_processing.go:74] worker 2 stopped. I0430 23:28:29.818484 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0430 23:28:29.818509 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0430 23:28:29.818543 1 recorder.go:75] Recording config/ingress with fingerprint=58fbc684c0e41cc896b9dc42d99bcba18ac5a5f368bd3bb8479c70e629693534 I0430 23:28:29.818553 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 7.570457ms to process 1 records W0430 23:28:29.818583 1 operator.go:328] started I0430 23:28:29.818607 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0430 23:28:29.822329 1 tasks_processing.go:74] worker 16 stopped. I0430 23:28:29.822348 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 11.516452ms to process 0 records I0430 23:28:29.822356 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 11.74284ms to process 0 records I0430 23:28:29.822362 1 tasks_processing.go:74] worker 0 stopped. I0430 23:28:29.822430 1 tasks_processing.go:74] worker 7 stopped. I0430 23:28:29.822439 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 11.756164ms to process 0 records I0430 23:28:29.822639 1 tasks_processing.go:74] worker 15 stopped. I0430 23:28:29.822810 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=4ad4400d4846d5910113d43a62221806fe3dab99878b3b06bbeabbcb418dc955 I0430 23:28:29.822917 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=3acc69543233e4a37d6311936ee494ccd84ccd82f25438bafca4566df0e36805 I0430 23:28:29.822945 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=28106a011b17b8eec267e007cf67eb580073deddc78ae5618e165aac079b7d1d I0430 23:28:29.822969 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=350636b94c425b9c6750bcb069cf1d531937b3d631f04d5633074bc3d68b00a0 I0430 23:28:29.823003 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=7dd52625cd4f0e78f083538edd0e546778755f1d8faed9b2d4c88c978175c59c I0430 23:28:29.823028 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=565f85fcc406744bb0056efdf61691f4cf84519bb90773b000336d44ce67869f I0430 23:28:29.823055 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=3f30d0fa2d37c11ccdde12c617f3550de70adfcd82643bb3bb3cf1cdaf9dafde I0430 23:28:29.823094 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=3297af29b56aaa9c57ab834eb55c40ed5a85703c50fac32e8c07591cf4b55d0f I0430 23:28:29.823117 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=bb59f23e21dae61d7231a6d2b97e7e6cd202da36b6cb73ba11979925338e58c3 I0430 23:28:29.823147 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=f8babaceffe6fa39a9c6798ea3030720e7740b4ac8cd79abf42cd602cee2467b I0430 23:28:29.823171 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=bcef550fddcec0c33c950616216f136e2899db0f09c97fe892f3f38ba503c495 I0430 23:28:29.823179 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 12.144448ms to process 11 records I0430 23:28:29.823253 1 recorder.go:75] Recording config/image with fingerprint=2ac8cf2923b4f2f9a2ebc55f706f59fbe9e3867b5b089fd836e71d0576c4c37b I0430 23:28:29.823258 1 tasks_processing.go:74] worker 41 stopped. I0430 23:28:29.823263 1 gather.go:177] gatherer "clusterconfig" function "image" took 11.481411ms to process 1 records E0430 23:28:29.823765 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host I0430 23:28:29.823781 1 tasks_processing.go:74] worker 54 stopped. E0430 23:28:29.823778 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host E0430 23:28:29.823787 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host I0430 23:28:29.823797 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 10.099384ms to process 0 records E0430 23:28:29.823807 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host E0430 23:28:29.823811 1 gather_active_alerts.go:64] Unable to retrieve most recent alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host I0430 23:28:29.823822 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 10.348702ms to process 0 records I0430 23:28:29.823827 1 tasks_processing.go:74] worker 5 stopped. E0430 23:28:29.823823 1 gather_silenced_alerts.go:52] Unable to retrieve silenced alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host E0430 23:28:29.823858 1 gather.go:140] gatherer "clusterconfig" function "active_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host I0430 23:28:29.823867 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 10.450047ms to process 0 records E0430 23:28:29.823875 1 gather.go:140] gatherer "clusterconfig" function "silenced_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: no such host I0430 23:28:29.823883 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 13.347032ms to process 0 records I0430 23:28:29.823890 1 tasks_processing.go:74] worker 6 stopped. I0430 23:28:29.823894 1 tasks_processing.go:74] worker 62 stopped. I0430 23:28:29.835518 1 tasks_processing.go:74] worker 32 stopped. I0430 23:28:29.835531 1 gather_logs.go:145] no pods in namespace were found I0430 23:28:29.835533 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 23.822436ms to process 0 records E0430 23:28:29.835538 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0430 23:28:29.835543 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 23.140553ms to process 0 records I0430 23:28:29.835550 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 24.877674ms to process 0 records I0430 23:28:29.835550 1 tasks_processing.go:74] worker 39 stopped. I0430 23:28:29.835555 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 24.748679ms to process 0 records I0430 23:28:29.835560 1 tasks_processing.go:74] worker 63 stopped. I0430 23:28:29.835566 1 tasks_processing.go:74] worker 11 stopped. I0430 23:28:29.835970 1 tasks_processing.go:74] worker 47 stopped. I0430 23:28:29.836156 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=b3e671b0bc6036799f1e269ca4d60825bcf20c973b4d55c54ccd7d40505d8b6b I0430 23:28:29.836173 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 25.459945ms to process 1 records I0430 23:28:29.843294 1 tasks_processing.go:74] worker 59 stopped. I0430 23:28:29.843309 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 32.396331ms to process 0 records I0430 23:28:29.847371 1 tasks_processing.go:74] worker 3 stopped. I0430 23:28:29.847386 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 36.034482ms to process 0 records I0430 23:28:29.847405 1 tasks_processing.go:74] worker 21 stopped. I0430 23:28:29.847410 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 33.463905ms to process 0 records I0430 23:28:29.847482 1 tasks_processing.go:74] worker 31 stopped. E0430 23:28:29.847497 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0430 23:28:29.847505 1 gather.go:177] gatherer "clusterconfig" function "machines" took 33.919046ms to process 0 records I0430 23:28:29.847512 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 34.174278ms to process 0 records I0430 23:28:29.847517 1 tasks_processing.go:74] worker 23 stopped. I0430 23:28:29.847527 1 tasks_processing.go:74] worker 57 stopped. I0430 23:28:29.847602 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=7a9baeeea6b0732c5013bffe8caa27e7d0137c1cea89de04b3e390582559e57f I0430 23:28:29.847626 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=302c9ea7bd767d9d13b3e994daa7096700948e5c2a42199f107fbbc7d88cbead I0430 23:28:29.847633 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 36.711008ms to process 2 records I0430 23:28:29.847666 1 tasks_processing.go:74] worker 37 stopped. I0430 23:28:29.847880 1 recorder.go:75] Recording config/network with fingerprint=dae270e109094890335be67ae22b1c7c5b15bcf2a219dd7097a2e57d89b80e20 I0430 23:28:29.847903 1 gather.go:177] gatherer "clusterconfig" function "networks" took 36.054348ms to process 1 records I0430 23:28:29.847991 1 tasks_processing.go:74] worker 22 stopped. I0430 23:28:29.848003 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 35.399852ms to process 0 records I0430 23:28:29.848009 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 37.351457ms to process 0 records I0430 23:28:29.848014 1 tasks_processing.go:74] worker 9 stopped. I0430 23:28:29.848106 1 tasks_processing.go:74] worker 53 stopped. I0430 23:28:29.848381 1 recorder.go:75] Recording config/oauth with fingerprint=e93f1bb34f20f9778488ea62aba42e4479c5c22ae3c251418303b58a0faba93f I0430 23:28:29.848402 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 33.647138ms to process 1 records I0430 23:28:29.848488 1 tasks_processing.go:74] worker 13 stopped. I0430 23:28:29.848598 1 recorder.go:75] Recording config/apiserver with fingerprint=83c36191b9b3a96751337942211677bf01f394e4dfe095c3920ca4dd517b5d53 I0430 23:28:29.848612 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 36.11902ms to process 1 records I0430 23:28:29.848620 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 34.684361ms to process 0 records I0430 23:28:29.848691 1 tasks_processing.go:74] worker 46 stopped. I0430 23:28:29.848729 1 tasks_processing.go:74] worker 60 stopped. I0430 23:28:29.850700 1 recorder.go:75] Recording config/infrastructure with fingerprint=9e3d56d06854e1845d8b746df05aaf4101c528b8062cec364f6a6fbe33faba3e I0430 23:28:29.850729 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 37.044432ms to process 1 records I0430 23:28:29.850782 1 tasks_processing.go:74] worker 51 stopped. I0430 23:28:29.850845 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=f6e53f827545e331a71abcb5ecf3b4516adfda455abce7d3be709dcc55cb4b77 I0430 23:28:29.851141 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 37.350195ms to process 1 records I0430 23:28:29.851160 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 35.668359ms to process 0 records I0430 23:28:29.851168 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 33.974922ms to process 0 records I0430 23:28:29.851176 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 34.697057ms to process 0 records I0430 23:28:29.851236 1 tasks_processing.go:74] worker 20 stopped. I0430 23:28:29.851244 1 tasks_processing.go:74] worker 35 stopped. I0430 23:28:29.851242 1 recorder.go:75] Recording config/proxy with fingerprint=42c6665e2d731ae0d1078592be76e9a1b1741ad0a4d744c14bb5855cdce422f7 I0430 23:28:29.851248 1 tasks_processing.go:74] worker 10 stopped. I0430 23:28:29.851262 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 37.003335ms to process 1 records I0430 23:28:29.851261 1 tasks_processing.go:74] worker 25 stopped. I0430 23:28:29.851342 1 tasks_processing.go:74] worker 34 stopped. I0430 23:28:29.851539 1 recorder.go:75] Recording config/node/ip-10-0-2-108.ec2.internal with fingerprint=19bf57e05fac353b26e658dcb3c5c6cc19a263d36827003dd7f8e8bcd065ed58 I0430 23:28:29.851550 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 37.146131ms to process 1 records I0430 23:28:29.851637 1 tasks_processing.go:74] worker 19 stopped. I0430 23:28:29.851711 1 recorder.go:75] Recording config/authentication with fingerprint=cc72c11c63145b06eda17ddda4e288f5824ed6b46d262ec16ae767bca0a95185 I0430 23:28:29.851720 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 36.417239ms to process 1 records I0430 23:28:29.851732 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 35.707899ms to process 0 records I0430 23:28:29.851751 1 tasks_processing.go:74] worker 29 stopped. I0430 23:28:29.851819 1 tasks_processing.go:74] worker 14 stopped. I0430 23:28:29.852140 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=89b2eab7ee54f1235f82c31722f1f40f57a86afcf19ac7e4d41c1578edc3face I0430 23:28:29.852154 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 37.283561ms to process 1 records I0430 23:28:29.853882 1 tasks_processing.go:74] worker 58 stopped. I0430 23:28:29.853900 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 40.192863ms to process 0 records I0430 23:28:29.856701 1 tasks_processing.go:74] worker 26 stopped. I0430 23:28:29.856802 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=5ac3d69255961aa60f820fcf90cb4f47d02670c26e351678ccdd8f869399244a I0430 23:28:29.856846 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=831434d1eaad9f7907077e510ca344ce99eb7a5d160233e60184dc9ca61caa63 I0430 23:28:29.856863 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=2ae57e03dfeff5e709248f1198c9abd564aede22a3f9ac56d5ed5991f7252cda I0430 23:28:29.856894 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=1c5460c71a0103e22b9a33907f7a7930b75f2c845035836e6a464e1ef3b41975 I0430 23:28:29.856913 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=8264f806b8e1cb53d709331ed4934bb9c0887b424ebe55256f4ccfc8fffcf9b3 I0430 23:28:29.856926 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=f8178f4f08641c2a04709bb0165793d4ee7ba42a2a9ec1568224dccd1e4e2b21 I0430 23:28:29.856933 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 44.216204ms to process 6 records I0430 23:28:29.856941 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 43.612476ms to process 0 records I0430 23:28:29.856954 1 tasks_processing.go:74] worker 18 stopped. I0430 23:28:29.857135 1 tasks_processing.go:74] worker 24 stopped. I0430 23:28:29.857246 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=8847042c4cff2aaa470351cdbd94fedb07a20f174597b472049d1fee870dd6bb I0430 23:28:29.857277 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=e3603e0efa204c14bd6fc3a6de910410fe513f0b53ca117967e9f18febd2f7ae I0430 23:28:29.857298 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=626397c3e9c1df24f96e1435279975a003c61522f3de6bd3f4edeffc8789cc1d I0430 23:28:29.857308 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 45.375929ms to process 3 records I0430 23:28:29.858476 1 tasks_processing.go:74] worker 12 stopped. I0430 23:28:29.858486 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 40.469559ms to process 0 records E0430 23:28:29.858494 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0430 23:28:29.858499 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 40.281607ms to process 0 records I0430 23:28:29.858502 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 43.743143ms to process 0 records I0430 23:28:29.858507 1 tasks_processing.go:74] worker 36 stopped. I0430 23:28:29.858506 1 tasks_processing.go:74] worker 1 stopped. I0430 23:28:29.858515 1 tasks_processing.go:74] worker 38 stopped. E0430 23:28:29.858524 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0430 23:28:29.858531 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 43.408539ms to process 0 records I0430 23:28:29.858548 1 tasks_processing.go:74] worker 43 stopped. I0430 23:28:29.858557 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 46.111138ms to process 0 records I0430 23:28:29.861228 1 tasks_processing.go:74] worker 27 stopped. I0430 23:28:29.861244 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 46.25313ms to process 0 records E0430 23:28:29.861255 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0430 23:28:29.861263 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 47.964782ms to process 0 records I0430 23:28:29.861270 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 46.147483ms to process 0 records I0430 23:28:29.861279 1 tasks_processing.go:74] worker 44 stopped. I0430 23:28:29.861285 1 tasks_processing.go:74] worker 55 stopped. I0430 23:28:29.862164 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0430 23:28:29.862177 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0430 23:28:29.862181 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0430 23:28:29.862185 1 controller.go:254] Source scaController *sca.Controller is not ready I0430 23:28:29.862189 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0430 23:28:29.862221 1 controller.go:531] The operator is still being initialized I0430 23:28:29.862231 1 controller.go:554] The operator is healthy I0430 23:28:29.865089 1 tasks_processing.go:74] worker 4 stopped. E0430 23:28:29.865103 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0430 23:28:29.865125 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0430 23:28:29.865136 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 54.051063ms to process 1 records I0430 23:28:29.866663 1 tasks_processing.go:74] worker 8 stopped. I0430 23:28:29.868051 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-66dbbb6468-vd2tm with fingerprint=7d1d7fe87dbadb74b1956b0df472b0c9c068cc4c230cc99373823b72361e61cf I0430 23:28:29.868117 1 recorder.go:75] Recording config/running_containers with fingerprint=5a4079c2fc7b4bbf8dac13b9da602b4f46bab41e9376ff80d4cd4b6f3fc81928 I0430 23:28:29.868127 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 56.036781ms to process 2 records I0430 23:28:29.868144 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0430 23:28:29.868151 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 53.768148ms to process 1 records I0430 23:28:29.868158 1 tasks_processing.go:74] worker 49 stopped. W0430 23:28:29.870993 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0430 23:28:29.874308 1 tasks_processing.go:74] worker 45 stopped. I0430 23:28:29.874324 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 62.985576ms to process 0 records I0430 23:28:29.876340 1 tasks_processing.go:74] worker 50 stopped. I0430 23:28:29.876357 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 62.06741ms to process 0 records I0430 23:28:29.878580 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0430 23:28:29.878588 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0430 23:28:29.878602 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0430 23:28:29.879638 1 tasks_processing.go:74] worker 33 stopped. I0430 23:28:29.879894 1 recorder.go:75] Recording config/version with fingerprint=81e66de87ce9fea077d2ee0cc9f95fcc4f0efc02a29521f45120607e543f9ea1 I0430 23:28:29.879913 1 recorder.go:75] Recording config/id with fingerprint=f6f07cb420475a5e196b10bb6d42367f7d800b6276998f5cfa7184beeea35609 I0430 23:28:29.879919 1 gather.go:177] gatherer "clusterconfig" function "version" took 65.072551ms to process 2 records I0430 23:28:29.880031 1 tasks_processing.go:74] worker 52 stopped. I0430 23:28:29.880506 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=8df48615abc95b8331e23da165ee53a1bbb9c9f186d342019b111069dce99a70 I0430 23:28:29.880715 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=c1256eae9b5e7bd1270c074befeaaeaa0faf9f2a142a11f880f823aea65449a4 I0430 23:28:29.880728 1 gather.go:177] gatherer "clusterconfig" function "crds" took 65.040351ms to process 2 records I0430 23:28:29.880810 1 tasks_processing.go:74] worker 17 stopped. I0430 23:28:29.880895 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=f4179a04bd2f943daaf82ccbc609e1b406f94cd55111dea910fe93eba8840e25 I0430 23:28:29.880982 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=e4e42e87c5a89fca6b810f763130746925f8597763381deb8730a6530a990e06 I0430 23:28:29.880989 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 66.486221ms to process 2 records I0430 23:28:29.882273 1 base_controller.go:82] Caches are synced for ConfigController I0430 23:28:29.882287 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0430 23:28:29.883362 1 tasks_processing.go:74] worker 30 stopped. I0430 23:28:29.883383 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 71.096735ms to process 0 records I0430 23:28:29.883932 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0430 23:28:29.898435 1 configmapobserver.go:84] configmaps "insights-config" not found I0430 23:28:29.906590 1 tasks_processing.go:74] worker 28 stopped. E0430 23:28:29.906608 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0430 23:28:29.906617 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0430 23:28:29.906622 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0430 23:28:29.906633 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0430 23:28:29.906670 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0430 23:28:29.906677 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0430 23:28:29.906682 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0430 23:28:29.906692 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0430 23:28:29.906729 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0430 23:28:29.906737 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0430 23:28:29.906743 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 95.581522ms to process 7 records I0430 23:28:29.919489 1 base_controller.go:82] Caches are synced for LoggingSyncer I0430 23:28:29.919502 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0430 23:28:29.953649 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0430 23:28:29.955965 1 cluster_transfer.go:102] no available accepted cluster transfer I0430 23:28:29.955981 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0430 23:28:29.957627 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0430 23:28:29.978995 1 tasks_processing.go:74] worker 42 stopped. E0430 23:28:29.979013 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0430 23:28:29.979019 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q0p22akbres55ngbqbdeuklsvdbep2e-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q0p22akbres55ngbqbdeuklsvdbep2e-primary-cert-bundle-secret" not found I0430 23:28:29.979074 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=54571b925741b7061abc55bb3fbc8d7b7d3055b2e92bd9470dee3ca00368d938 I0430 23:28:29.979088 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 168.319999ms to process 1 records I0430 23:28:30.055735 1 sca.go:266] etc-pki-entitlement secret successfully created I0430 23:28:30.055752 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated W0430 23:28:30.869588 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0430 23:28:31.870406 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0430 23:28:32.294072 1 tasks_processing.go:74] worker 48 stopped. I0430 23:28:32.294192 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=9590b5bdd00736a60b20dba3ce412de1c042021b031abca6183e9cb7330ccb5d I0430 23:28:32.294241 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=b1d3a57201f472fba248e6e22732d03f4ebac77f3d7204455295e9c6f31f376a I0430 23:28:32.294270 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=1412eae0dcb4c4c4964e241ca5d88ed2b8ae618c518101b34c0737f0a81d585a I0430 23:28:32.294280 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0430 23:28:32.294317 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=265ba2aefcc6171e2aba42f2e47740ae125005bf59635bc5f63be2b35b967961 I0430 23:28:32.294332 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0430 23:28:32.294352 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=93f323383a4470e549f78ee5e69eb885c7cc3e40895b6a90c8b1a60ae6639147 I0430 23:28:32.294379 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=fc4bb256c92ed47c0a68f6519dfc42cb7c4f9dc46e65904f62ed8ac6e6290184 I0430 23:28:32.294402 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=735e9adaaa93e3db84af2e86678c3d9c5f0210a0f51453ab80d62e778899d70b I0430 23:28:32.294433 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=e8bdeafdec0c26c0289b0bcf15c10bb03ec5500573cc8360c32d904a7375832e I0430 23:28:32.294693 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=505a50f6703a84c0aeceaeebdc191c8f743c966e53d4262676ffc28c97d8b395 I0430 23:28:32.294714 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=caf82902754d8a915ebde16f7f8d12456427beb43e9f9ae8518f8035bc7be460 I0430 23:28:32.294723 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0430 23:28:32.294739 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=7c500f6d30bc6ba3de6ac650a7db60d575f49f1daccd25c81bde9f1a84cdaa43 I0430 23:28:32.294749 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0430 23:28:32.294765 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=70cca38c41adce9ac159e7d90246c4e428ee316ea3fa9d8eb1bf2632ae3f1dce I0430 23:28:32.294775 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0430 23:28:32.294794 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=98e132442ce38f4fa12e5021b1eb587e0e9604e8e2a350b09a802d852a3e4dbb I0430 23:28:32.294804 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0430 23:28:32.294823 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=0b24d5f5e5138bb886b2da134ea6d8b55d80222126cc38e4e8ca91e80e218868 I0430 23:28:32.294941 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=d698ab0d3d06c0899f08e73fb3487c5545106f3282b7bc98afeb24b24b94088c I0430 23:28:32.294951 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0430 23:28:32.294958 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0430 23:28:32.294978 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0430 23:28:32.295002 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=2b7beae6f849c5a4ba7711b1e3868263aaa344ff201c4936884dfa2cce3a9488 I0430 23:28:32.295027 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=f95499fae922f0e0311ca1cbe249560819bc44977114c74e1adbe774ac0adffc I0430 23:28:32.295037 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0430 23:28:32.295053 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=b09ffba57328f29527e04731aa86921866c11180686aca7fdf2a23b0807ffd20 I0430 23:28:32.295063 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0430 23:28:32.295080 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=9df70dc9d45ffa30fd3e1baacd9b855cd150735ca737a0fbed3f0f45b13ad1f9 I0430 23:28:32.295088 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0430 23:28:32.295100 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=d89ce83206c74d8fcb2fbbaf2a813b54a9d822485e3754f2a5c04fd1ce590982 I0430 23:28:32.295115 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=259f5188804e32be23d62899cc558470189e022e463b163e450cebadf6076472 I0430 23:28:32.295129 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=3f3c7cc0762a2cf787f48f1a5bb39a12936291a4754c587fbf70bc619a7652cc I0430 23:28:32.295150 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=28205888b4e26670ef004a65f80193ffbb49bf014ea5be860268055eab0f8eb4 I0430 23:28:32.295166 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0430 23:28:32.295198 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=795f0c78d4f6eb96699413e58c593678497ecd4c7582f2a4e8fbef412b1dd6fd I0430 23:28:32.295222 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0430 23:28:32.295233 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0430 23:28:32.295240 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.480498295s to process 39 records W0430 23:28:32.869826 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0430 23:28:33.870380 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0430 23:28:34.867904 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0430 23:28:34.867930 1 tasks_processing.go:74] worker 40 stopped. E0430 23:28:34.867941 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0430 23:28:34.867956 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0430 23:28:34.867969 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.05296428s to process 1 records I0430 23:28:42.663460 1 tasks_processing.go:74] worker 61 stopped. I0430 23:28:42.663501 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0430 23:28:42.663513 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.85187857s to process 1 records I0430 23:28:43.422450 1 tasks_processing.go:74] worker 56 stopped. I0430 23:28:43.422715 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=082aca9dc3733be58f521006db7d656935c483be78b436c19f7b798fedba354a I0430 23:28:43.422731 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.610610918s to process 1 records E0430 23:28:43.422797 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.612s with: function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"active_alerts\" failed with an error, function \"silenced_alerts\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0430 23:28:43.423911 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "tsdb_status" failed with an error, function "metrics" failed with an error, function "active_alerts" failed with an error, function "silenced_alerts" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "machine_healthchecks" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0430 23:28:43.423927 1 periodic.go:216] Running workloads gatherer I0430 23:28:43.423942 1 tasks_processing.go:45] number of workers: 2 I0430 23:28:43.423952 1 tasks_processing.go:69] worker 1 listening for tasks. I0430 23:28:43.423957 1 tasks_processing.go:71] worker 1 working on workload_info task. I0430 23:28:43.423967 1 tasks_processing.go:69] worker 0 listening for tasks. I0430 23:28:43.423986 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0430 23:28:43.430115 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-108.ec2.internal... I0430 23:28:43.448157 1 tasks_processing.go:74] worker 0 stopped. I0430 23:28:43.448177 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 24.153283ms to process 0 records I0430 23:28:47.006942 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 4s I0430 23:28:47.020935 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 19s for image data I0430 23:28:47.030912 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (10ms) I0430 23:28:47.039977 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (9ms) I0430 23:28:47.048701 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (9ms) I0430 23:28:47.057282 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (9ms) I0430 23:28:47.065405 1 gather_workloads_info.go:387] No image sha256:15e1b5f763553e86d0230a751b0783e3b990438fd07cf96eed164bccb72ed564 (8ms) I0430 23:28:47.073342 1 gather_workloads_info.go:387] No image sha256:bdeb1c455cffccdb6c90db238e9d371de06340b1d6105d47f66d87293f878ac1 (8ms) I0430 23:28:47.081199 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (8ms) I0430 23:28:47.092365 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (11ms) I0430 23:28:47.100817 1 gather_workloads_info.go:387] No image sha256:5ff204630794311b0b37fa7b197b933ad85d76a481bf7cdb3bcbada08f0cdcbf (8ms) I0430 23:28:47.109147 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (8ms) I0430 23:28:47.129744 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (21ms) I0430 23:28:47.230823 1 gather_workloads_info.go:398] Found image sha256:23204877f166455ce8f9de9b91a467eeda576f172d5f96124a68aa986c4775d6 (101ms) I0430 23:28:47.331506 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (100ms) I0430 23:28:47.430164 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (99ms) I0430 23:28:47.530948 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (101ms) I0430 23:28:47.630721 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (100ms) I0430 23:28:47.730996 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (100ms) I0430 23:28:47.830332 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (99ms) I0430 23:28:47.930923 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (101ms) I0430 23:28:48.030737 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (100ms) I0430 23:28:48.130562 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (100ms) I0430 23:28:48.230675 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (100ms) I0430 23:28:48.331020 1 gather_workloads_info.go:387] No image sha256:572ebf7c7fab1a518582f3ca46c866730423cce6e8a19f614dc15fd61bf0c65a (100ms) I0430 23:28:48.430557 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (100ms) I0430 23:28:48.530150 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (100ms) I0430 23:28:48.630695 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (101ms) I0430 23:28:48.731868 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (101ms) I0430 23:28:48.830234 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (98ms) I0430 23:28:48.930751 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (100ms) I0430 23:28:49.030926 1 gather_workloads_info.go:398] Found image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (100ms) I0430 23:28:49.131033 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (100ms) I0430 23:28:49.235107 1 gather_workloads_info.go:387] No image sha256:5940cc34b322325618929b1e0eb3864dae73d54c8f5b474e218dfb28123a61c7 (104ms) I0430 23:28:49.370114 1 gather_workloads_info.go:387] No image sha256:0b2e6536a0ca3afc5e64a218087a6287c74b77a359075a385f1949ca5bd51b22 (135ms) I0430 23:28:49.432005 1 gather_workloads_info.go:387] No image sha256:6c6d412ccf0c3d6a14645a79df6c4658c0f8725d7954cb83c3d2f0116499345a (62ms) I0430 23:28:49.531083 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (99ms) I0430 23:28:49.531119 1 tasks_processing.go:74] worker 1 stopped. I0430 23:28:49.531332 1 recorder.go:75] Recording config/workload_info with fingerprint=245c062d289c7b9c5e4b7abdb82196761c68ac18f8e3b4ddb463fd5e2632e791 I0430 23:28:49.531348 1 gather.go:177] gatherer "workloads" function "workload_info" took 6.107155646s to process 1 records I0430 23:28:49.531362 1 periodic.go:263] Periodic gather workloads completed in 6.107s I0430 23:28:49.531370 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0430 23:28:49.531375 1 periodic.go:216] Running conditional gatherer I0430 23:28:49.537515 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0430 23:28:49.713614 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer E0430 23:28:49.716101 1 conditional_gatherer.go:324] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: no such host I0430 23:28:49.716177 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0430 23:28:49.722115 1 conditional_gatherer.go:392] cluster version is '4.21.9' E0430 23:28:49.722130 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0430 23:28:49.722136 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0430 23:28:49.722140 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0430 23:28:49.722143 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0430 23:28:49.722145 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0430 23:28:49.722149 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0430 23:28:49.722152 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0430 23:28:49.722154 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0430 23:28:49.722157 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0430 23:28:49.722161 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0430 23:28:49.722174 1 tasks_processing.go:45] number of workers: 3 I0430 23:28:49.722186 1 tasks_processing.go:69] worker 2 listening for tasks. I0430 23:28:49.722190 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0430 23:28:49.722198 1 tasks_processing.go:69] worker 0 listening for tasks. I0430 23:28:49.722213 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0430 23:28:49.722212 1 tasks_processing.go:69] worker 1 listening for tasks. I0430 23:28:49.722220 1 tasks_processing.go:71] worker 0 working on conditional_gatherer_rules task. I0430 23:28:49.722226 1 tasks_processing.go:74] worker 1 stopped. I0430 23:28:49.722297 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0430 23:28:49.722327 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.631µs to process 1 records I0430 23:28:49.722409 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=9fafdcbf373f1f367f1d69551ea59114592cce1681189be0f3dde774e0530661 I0430 23:28:49.722415 1 tasks_processing.go:74] worker 0 stopped. I0430 23:28:49.722422 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 507ns to process 1 records I0430 23:28:49.722433 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 214.071µs to process 0 records I0430 23:28:49.722443 1 tasks_processing.go:74] worker 2 stopped. I0430 23:28:49.722464 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0430 23:28:49.722490 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0430 23:28:49.748477 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=72372d3c5de2da8f6d0ea3185486143280c0a090d74b9e2e361b254f52a73f86 I0430 23:28:49.748609 1 diskrecorder.go:70] Writing 99 records to /var/lib/insights-operator/insights-2026-04-30-232849.tar.gz I0430 23:28:49.754723 1 diskrecorder.go:51] Wrote 99 records to disk in 6ms I0430 23:28:49.754755 1 periodic.go:285] Gathering cluster info every 2h0m0s I0430 23:28:49.754771 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0430 23:29:59.811438 1 diskrecorder.go:223] Found files to send: insights-2026-04-30-232849.tar.gz I0430 23:29:59.811473 1 insightsuploader.go:150] Checking archives to upload periodically every 16m22.869421595s I0430 23:29:59.811482 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0430 23:29:59.825060 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0430 23:30:00.070994 1 requests.go:87] Successfully reported id=2026-04-30T23:29:59Z x-rh-insights-request-id=8f409cfaa10744bcb7d790002b81f418, wrote=45385 I0430 23:30:00.071131 1 insightsuploader.go:187] Uploaded report successfully in 259.638107ms I0430 23:30:00.071158 1 controller.go:129] Initializing last reported time to 2026-04-30T23:29:59Z I0430 23:30:00.071232 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0430 23:30:00.071245 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0430 23:30:00.071253 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0430 23:30:00.078111 1 controller.go:554] The operator is healthy I0430 23:30:29.825826 1 controller.go:554] The operator is healthy I0430 23:31:02.832598 1 insightsreport.go:137] Pulling report from smart-proxy I0430 23:31:02.832628 1 insightsreport.go:149] Retrieving report I0430 23:31:02.839645 1 requests.go:110] Retrieving report for cluster: 45481509-02b9-4247-bfb3-ccebc1a05e46 I0430 23:31:02.839658 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/45481509-02b9-4247-bfb3-ccebc1a05e46/reports I0430 23:31:02.842761 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/45481509-02b9-4247-bfb3-ccebc1a05e46/reports I0430 23:31:03.088407 1 insightsreport.go:184] Report retrieved I0430 23:31:03.117496 1 insightsreport.go:239] Report retrieved correctly I0430 23:32:29.825975 1 controller.go:554] The operator is healthy I0430 23:33:29.804137 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0430 23:33:29.809775 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0430 23:33:29.809791 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0430 23:33:29.813712 1 secretconfigobserver.go:119] support secret does not exist I0430 23:34:29.825021 1 controller.go:554] The operator is healthy I0430 23:36:29.829124 1 controller.go:554] The operator is healthy I0430 23:38:29.798383 1 configmapobserver.go:84] configmaps "insights-config" not found I0430 23:38:29.814518 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0430 23:38:29.819195 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0430 23:38:29.819211 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0430 23:38:29.823738 1 secretconfigobserver.go:119] support secret does not exist I0430 23:38:29.824289 1 controller.go:554] The operator is healthy I0430 23:40:29.826464 1 controller.go:554] The operator is healthy I0430 23:42:29.824795 1 controller.go:554] The operator is healthy I0430 23:43:29.824256 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0430 23:43:29.829726 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0430 23:43:29.829749 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0430 23:43:29.833759 1 secretconfigobserver.go:119] support secret does not exist I0430 23:44:29.824923 1 controller.go:554] The operator is healthy I0430 23:46:27.941726 1 insightsuploader.go:146] Nothing to report since 2026-04-30T23:29:59Z I0430 23:46:29.826287 1 controller.go:554] The operator is healthy I0430 23:48:29.799327 1 configmapobserver.go:84] configmaps "insights-config" not found I0430 23:48:29.825417 1 controller.go:554] The operator is healthy I0430 23:48:29.834250 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0430 23:48:29.839191 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0430 23:48:29.839208 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0430 23:48:29.843192 1 secretconfigobserver.go:119] support secret does not exist I0430 23:50:29.829776 1 controller.go:554] The operator is healthy I0430 23:52:29.829535 1 controller.go:554] The operator is healthy I0430 23:53:29.843718 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0430 23:53:29.852990 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0430 23:53:29.853010 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0430 23:53:29.857648 1 secretconfigobserver.go:119] support secret does not exist I0430 23:54:29.827985 1 controller.go:554] The operator is healthy I0430 23:56:29.833158 1 controller.go:554] The operator is healthy I0430 23:58:29.799329 1 configmapobserver.go:84] configmaps "insights-config" not found I0430 23:58:29.825127 1 controller.go:554] The operator is healthy I0430 23:58:29.858127 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0430 23:58:29.863014 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0430 23:58:29.863033 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0430 23:58:29.867010 1 secretconfigobserver.go:119] support secret does not exist I0501 00:00:29.825223 1 controller.go:554] The operator is healthy