W0416 19:17:33.055877 1 cmd.go:257] Using insecure, self-signed certificates I0416 19:17:33.399536 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0416 19:17:33.399931 1 observer_polling.go:159] Starting file observer I0416 19:17:33.903621 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0416 19:17:33.903799 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0416 19:17:33.904370 1 secure_serving.go:57] Forcing use of http/1.1 only W0416 19:17:33.904395 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0416 19:17:33.904400 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0416 19:17:33.904404 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0416 19:17:33.904406 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0416 19:17:33.904410 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0416 19:17:33.904413 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0416 19:17:33.904456 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0416 19:17:33.908405 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0416 19:17:33.908445 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"d4e61c0a-4b3b-44a0-a55a-53ff6ee97bcb", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0416 19:17:33.908920 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0416 19:17:33.908936 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0416 19:17:33.908943 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0416 19:17:33.908944 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0416 19:17:33.908980 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0416 19:17:33.908971 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0416 19:17:33.909212 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-1058509088/tls.crt::/tmp/serving-cert-1058509088/tls.key" I0416 19:17:33.909554 1 secure_serving.go:213] Serving securely on [::]:8443 I0416 19:17:33.909581 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0416 19:17:33.913318 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0416 19:17:33.913342 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0416 19:17:33.913443 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0416 19:17:33.918166 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0416 19:17:33.918189 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0416 19:17:33.923327 1 secretconfigobserver.go:119] support secret does not exist I0416 19:17:33.927254 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0416 19:17:33.931024 1 secretconfigobserver.go:119] support secret does not exist I0416 19:17:33.933756 1 recorder.go:161] Pruning old reports every 5h6m25s, max age is 288h0m0s I0416 19:17:33.939621 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0416 19:17:33.939637 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0416 19:17:33.939664 1 periodic.go:209] Running clusterconfig gatherer I0416 19:17:33.939669 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0416 19:17:33.939688 1 insightsreport.go:296] Starting report retriever I0416 19:17:33.939696 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0416 19:17:33.939710 1 tasks_processing.go:45] number of workers: 64 I0416 19:17:33.939731 1 tasks_processing.go:69] worker 1 listening for tasks. I0416 19:17:33.939753 1 tasks_processing.go:69] worker 0 listening for tasks. I0416 19:17:33.939755 1 tasks_processing.go:71] worker 1 working on active_alerts task. I0416 19:17:33.939759 1 tasks_processing.go:71] worker 0 working on version task. I0416 19:17:33.939782 1 tasks_processing.go:69] worker 32 listening for tasks. W0416 19:17:33.939787 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0416 19:17:33.939793 1 tasks_processing.go:69] worker 16 listening for tasks. I0416 19:17:33.939797 1 tasks_processing.go:69] worker 17 listening for tasks. I0416 19:17:33.939801 1 tasks_processing.go:69] worker 48 listening for tasks. I0416 19:17:33.939804 1 tasks_processing.go:69] worker 2 listening for tasks. I0416 19:17:33.939804 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 37.369µs to process 0 records I0416 19:17:33.939808 1 tasks_processing.go:69] worker 18 listening for tasks. I0416 19:17:33.939810 1 tasks_processing.go:69] worker 33 listening for tasks. I0416 19:17:33.939817 1 tasks_processing.go:69] worker 49 listening for tasks. I0416 19:17:33.939813 1 tasks_processing.go:69] worker 3 listening for tasks. I0416 19:17:33.939820 1 tasks_processing.go:69] worker 4 listening for tasks. I0416 19:17:33.939824 1 tasks_processing.go:69] worker 19 listening for tasks. I0416 19:17:33.939814 1 tasks_processing.go:69] worker 41 listening for tasks. I0416 19:17:33.939831 1 tasks_processing.go:69] worker 5 listening for tasks. I0416 19:17:33.939829 1 tasks_processing.go:69] worker 44 listening for tasks. I0416 19:17:33.939836 1 tasks_processing.go:69] worker 20 listening for tasks. I0416 19:17:33.939836 1 tasks_processing.go:69] worker 45 listening for tasks. I0416 19:17:33.939820 1 tasks_processing.go:69] worker 42 listening for tasks. I0416 19:17:33.939831 1 tasks_processing.go:69] worker 40 listening for tasks. I0416 19:17:33.939845 1 tasks_processing.go:69] worker 21 listening for tasks. I0416 19:17:33.939843 1 tasks_processing.go:69] worker 34 listening for tasks. I0416 19:17:33.939846 1 tasks_processing.go:69] worker 57 listening for tasks. I0416 19:17:33.939851 1 tasks_processing.go:69] worker 12 listening for tasks. I0416 19:17:33.939863 1 tasks_processing.go:69] worker 22 listening for tasks. I0416 19:17:33.939825 1 tasks_processing.go:69] worker 43 listening for tasks. I0416 19:17:33.939859 1 tasks_processing.go:69] worker 11 listening for tasks. I0416 19:17:33.939866 1 tasks_processing.go:69] worker 38 listening for tasks. I0416 19:17:33.939875 1 tasks_processing.go:69] worker 63 listening for tasks. I0416 19:17:33.939880 1 tasks_processing.go:69] worker 25 listening for tasks. I0416 19:17:33.939844 1 tasks_processing.go:69] worker 6 listening for tasks. I0416 19:17:33.939844 1 tasks_processing.go:69] worker 59 listening for tasks. I0416 19:17:33.939886 1 tasks_processing.go:69] worker 7 listening for tasks. I0416 19:17:33.939883 1 tasks_processing.go:69] worker 37 listening for tasks. I0416 19:17:33.939892 1 tasks_processing.go:69] worker 29 listening for tasks. I0416 19:17:33.939894 1 tasks_processing.go:69] worker 53 listening for tasks. I0416 19:17:33.939897 1 tasks_processing.go:69] worker 36 listening for tasks. I0416 19:17:33.939898 1 tasks_processing.go:69] worker 10 listening for tasks. I0416 19:17:33.939902 1 tasks_processing.go:69] worker 31 listening for tasks. I0416 19:17:33.939903 1 tasks_processing.go:69] worker 30 listening for tasks. I0416 19:17:33.939887 1 tasks_processing.go:69] worker 55 listening for tasks. I0416 19:17:33.939850 1 tasks_processing.go:69] worker 46 listening for tasks. I0416 19:17:33.939851 1 tasks_processing.go:69] worker 60 listening for tasks. I0416 19:17:33.939860 1 tasks_processing.go:69] worker 35 listening for tasks. I0416 19:17:33.939860 1 tasks_processing.go:69] worker 61 listening for tasks. I0416 19:17:33.939837 1 tasks_processing.go:69] worker 58 listening for tasks. I0416 19:17:33.939830 1 tasks_processing.go:69] worker 50 listening for tasks. I0416 19:17:33.939868 1 tasks_processing.go:69] worker 62 listening for tasks. I0416 19:17:33.939870 1 tasks_processing.go:69] worker 23 listening for tasks. I0416 19:17:33.939869 1 tasks_processing.go:69] worker 13 listening for tasks. I0416 19:17:33.939874 1 tasks_processing.go:69] worker 24 listening for tasks. I0416 19:17:33.939877 1 tasks_processing.go:69] worker 51 listening for tasks. I0416 19:17:33.939921 1 tasks_processing.go:69] worker 27 listening for tasks. I0416 19:17:33.939876 1 tasks_processing.go:69] worker 14 listening for tasks. I0416 19:17:33.939877 1 tasks_processing.go:69] worker 47 listening for tasks. I0416 19:17:33.939877 1 tasks_processing.go:69] worker 39 listening for tasks. I0416 19:17:33.939885 1 tasks_processing.go:69] worker 9 listening for tasks. I0416 19:17:33.939885 1 tasks_processing.go:69] worker 52 listening for tasks. I0416 19:17:33.939886 1 tasks_processing.go:69] worker 54 listening for tasks. I0416 19:17:33.939888 1 tasks_processing.go:69] worker 28 listening for tasks. I0416 19:17:33.939890 1 tasks_processing.go:69] worker 8 listening for tasks. I0416 19:17:33.939895 1 tasks_processing.go:69] worker 26 listening for tasks. I0416 19:17:33.939967 1 tasks_processing.go:71] worker 42 working on mutating_webhook_configurations task. I0416 19:17:33.939893 1 tasks_processing.go:69] worker 56 listening for tasks. I0416 19:17:33.939973 1 tasks_processing.go:71] worker 33 working on service_accounts task. I0416 19:17:33.939976 1 tasks_processing.go:71] worker 12 working on node_logs task. I0416 19:17:33.939977 1 tasks_processing.go:71] worker 34 working on openstack_dataplanenodesets task. I0416 19:17:33.939979 1 tasks_processing.go:71] worker 56 working on operators task. I0416 19:17:33.939897 1 tasks_processing.go:69] worker 15 listening for tasks. I0416 19:17:33.939986 1 tasks_processing.go:71] worker 17 working on validating_webhook_configurations task. I0416 19:17:33.939965 1 tasks_processing.go:71] worker 4 working on ingress task. I0416 19:17:33.939997 1 tasks_processing.go:71] worker 5 working on crds task. I0416 19:17:33.940007 1 tasks_processing.go:71] worker 19 working on container_runtime_configs task. I0416 19:17:33.939973 1 tasks_processing.go:71] worker 26 working on olm_operators task. I0416 19:17:33.940023 1 tasks_processing.go:71] worker 37 working on nodes task. I0416 19:17:33.940036 1 tasks_processing.go:71] worker 7 working on monitoring_persistent_volumes task. I0416 19:17:33.940054 1 tasks_processing.go:71] worker 51 working on config_maps task. I0416 19:17:33.939968 1 tasks_processing.go:71] worker 57 working on metrics task. W0416 19:17:33.940092 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0416 19:17:33.940106 1 tasks_processing.go:71] worker 31 working on storage_classes task. I0416 19:17:33.940108 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 24.541µs to process 0 records I0416 19:17:33.940117 1 tasks_processing.go:71] worker 49 working on overlapping_namespace_uids task. I0416 19:17:33.939968 1 tasks_processing.go:71] worker 2 working on openstack_controlplanes task. I0416 19:17:33.940193 1 tasks_processing.go:71] worker 10 working on nodenetworkconfigurationpolicies task. I0416 19:17:33.940210 1 tasks_processing.go:71] worker 27 working on oauths task. I0416 19:17:33.939979 1 tasks_processing.go:71] worker 1 working on support_secret task. I0416 19:17:33.940251 1 tasks_processing.go:71] worker 50 working on pod_network_connectivity_checks task. I0416 19:17:33.940322 1 tasks_processing.go:74] worker 57 stopped. I0416 19:17:33.940335 1 tasks_processing.go:71] worker 15 working on storage_cluster task. I0416 19:17:33.940408 1 tasks_processing.go:71] worker 14 working on image task. I0416 19:17:33.940423 1 tasks_processing.go:71] worker 58 working on install_plans task. I0416 19:17:33.940478 1 tasks_processing.go:71] worker 30 working on openshift_machine_api_events task. I0416 19:17:33.940495 1 tasks_processing.go:71] worker 53 working on cluster_apiserver task. I0416 19:17:33.940520 1 tasks_processing.go:71] worker 22 working on operators_pods_and_events task. I0416 19:17:33.940524 1 tasks_processing.go:71] worker 29 working on openshift_logging task. I0416 19:17:33.940636 1 tasks_processing.go:71] worker 3 working on cost_management_metrics_configs task. I0416 19:17:33.940202 1 tasks_processing.go:71] worker 9 working on openstack_version task. I0416 19:17:33.940826 1 tasks_processing.go:71] worker 23 working on machines task. I0416 19:17:33.940879 1 tasks_processing.go:71] worker 54 working on machine_sets task. I0416 19:17:33.940924 1 tasks_processing.go:71] worker 24 working on ingress_certificates task. I0416 19:17:33.939971 1 tasks_processing.go:71] worker 18 working on dvo_metrics task. I0416 19:17:33.940829 1 tasks_processing.go:71] worker 44 working on sap_pods task. I0416 19:17:33.940902 1 tasks_processing.go:71] worker 36 working on image_pruners task. I0416 19:17:33.939974 1 tasks_processing.go:71] worker 32 working on sap_datahubs task. I0416 19:17:33.941263 1 tasks_processing.go:71] worker 61 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0416 19:17:33.939983 1 tasks_processing.go:71] worker 16 working on infrastructures task. I0416 19:17:33.940017 1 tasks_processing.go:71] worker 60 working on networks task. I0416 19:17:33.939969 1 tasks_processing.go:71] worker 48 working on schedulers task. I0416 19:17:33.940834 1 tasks_processing.go:71] worker 20 working on machine_config_pools task. I0416 19:17:33.940841 1 tasks_processing.go:71] worker 45 working on proxies task. I0416 19:17:33.940865 1 tasks_processing.go:71] worker 35 working on openstack_dataplanedeployments task. I0416 19:17:33.940873 1 tasks_processing.go:71] worker 62 working on machine_healthchecks task. I0416 19:17:33.939974 1 tasks_processing.go:71] worker 21 working on certificate_signing_requests task. I0416 19:17:33.940908 1 tasks_processing.go:71] worker 47 working on container_images task. I0416 19:17:33.940914 1 tasks_processing.go:71] worker 13 working on authentication task. I0416 19:17:33.940915 1 tasks_processing.go:71] worker 52 working on machine_configs task. I0416 19:17:33.940923 1 tasks_processing.go:71] worker 41 working on sap_config task. I0416 19:17:33.940922 1 tasks_processing.go:71] worker 39 working on machine_autoscalers task. I0416 19:17:33.940928 1 tasks_processing.go:71] worker 28 working on clusterroles task. I0416 19:17:33.940935 1 tasks_processing.go:71] worker 63 working on image_registries task. I0416 19:17:33.940934 1 tasks_processing.go:71] worker 8 working on qemu_kubevirt_launcher_logs task. I0416 19:17:33.940941 1 tasks_processing.go:71] worker 43 working on ceph_cluster task. I0416 19:17:33.940941 1 tasks_processing.go:71] worker 6 working on lokistack task. I0416 19:17:33.940945 1 tasks_processing.go:71] worker 11 working on pdbs task. I0416 19:17:33.939972 1 tasks_processing.go:71] worker 40 working on jaegers task. I0416 19:17:33.940949 1 tasks_processing.go:71] worker 38 working on tsdb_status task. I0416 19:17:33.940949 1 tasks_processing.go:71] worker 25 working on feature_gates task. W0416 19:17:33.942438 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0416 19:17:33.942450 1 tasks_processing.go:74] worker 38 stopped. I0416 19:17:33.942462 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 31.055µs to process 0 records I0416 19:17:33.940957 1 tasks_processing.go:71] worker 46 working on nodenetworkstates task. I0416 19:17:33.940954 1 tasks_processing.go:71] worker 59 working on aggregated_monitoring_cr_names task. I0416 19:17:33.940960 1 tasks_processing.go:71] worker 55 working on silenced_alerts task. W0416 19:17:33.942866 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0416 19:17:33.942879 1 tasks_processing.go:74] worker 55 stopped. I0416 19:17:33.942921 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 83.34µs to process 0 records I0416 19:17:33.944458 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0416 19:17:33.944476 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0416 19:17:33.944482 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0416 19:17:33.944485 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0416 19:17:33.944499 1 controller.go:489] The operator is still being initialized I0416 19:17:33.944505 1 controller.go:512] The operator is healthy I0416 19:17:33.945017 1 tasks_processing.go:74] worker 50 stopped. E0416 19:17:33.945028 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0416 19:17:33.945036 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 4.756681ms to process 0 records I0416 19:17:33.945068 1 tasks_processing.go:74] worker 19 stopped. I0416 19:17:33.945083 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 5.045946ms to process 0 records I0416 19:17:33.945751 1 tasks_processing.go:74] worker 15 stopped. I0416 19:17:33.945760 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 5.410483ms to process 0 records I0416 19:17:33.946936 1 tasks_processing.go:74] worker 34 stopped. I0416 19:17:33.946949 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 6.945134ms to process 0 records I0416 19:17:33.951469 1 tasks_processing.go:74] worker 3 stopped. I0416 19:17:33.951480 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 10.809979ms to process 0 records I0416 19:17:33.951908 1 tasks_processing.go:74] worker 23 stopped. E0416 19:17:33.951920 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0416 19:17:33.951927 1 gather.go:177] gatherer "clusterconfig" function "machines" took 11.021311ms to process 0 records I0416 19:17:33.955105 1 tasks_processing.go:74] worker 2 stopped. I0416 19:17:33.955122 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 14.902288ms to process 0 records I0416 19:17:33.956987 1 tasks_processing.go:74] worker 9 stopped. I0416 19:17:33.957004 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 16.267087ms to process 0 records I0416 19:17:33.957011 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 14.709979ms to process 0 records I0416 19:17:33.957015 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 14.653469ms to process 0 records I0416 19:17:33.957019 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 14.981931ms to process 0 records I0416 19:17:33.957022 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 15.065926ms to process 0 records I0416 19:17:33.957027 1 tasks_processing.go:74] worker 41 stopped. I0416 19:17:33.957030 1 tasks_processing.go:74] worker 6 stopped. I0416 19:17:33.957033 1 tasks_processing.go:74] worker 40 stopped. I0416 19:17:33.957037 1 tasks_processing.go:74] worker 39 stopped. I0416 19:17:33.957040 1 tasks_processing.go:74] worker 4 stopped. I0416 19:17:33.957256 1 recorder.go:75] Recording config/ingress with fingerprint=aa0e5a415995a0afa9b18d9be9783b33304e66988c60facfc62ca3d5ef5044f4 I0416 19:17:33.957275 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 17.036834ms to process 1 records I0416 19:17:33.958048 1 tasks_processing.go:74] worker 54 stopped. I0416 19:17:33.958059 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 17.113869ms to process 0 records I0416 19:17:33.958357 1 tasks_processing.go:74] worker 32 stopped. I0416 19:17:33.958368 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 17.103873ms to process 0 records I0416 19:17:33.958376 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 17.878722ms to process 0 records I0416 19:17:33.958381 1 tasks_processing.go:74] worker 30 stopped. I0416 19:17:33.958878 1 tasks_processing.go:74] worker 43 stopped. I0416 19:17:33.958888 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 16.671125ms to process 0 records I0416 19:17:33.958897 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 17.290193ms to process 0 records I0416 19:17:33.958902 1 tasks_processing.go:74] worker 35 stopped. I0416 19:17:33.958969 1 tasks_processing.go:74] worker 62 stopped. E0416 19:17:33.958981 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0416 19:17:33.958990 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 17.325928ms to process 0 records I0416 19:17:33.959077 1 tasks_processing.go:74] worker 27 stopped. I0416 19:17:33.959428 1 recorder.go:75] Recording config/oauth with fingerprint=9648dc9bd4c407664f39f099103125b2de79b6e8931d3d64baec874bf4ea6461 I0416 19:17:33.959446 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 18.851029ms to process 1 records I0416 19:17:33.959454 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 18.808893ms to process 0 records I0416 19:17:33.959462 1 tasks_processing.go:74] worker 29 stopped. I0416 19:17:33.959645 1 tasks_processing.go:74] worker 10 stopped. I0416 19:17:33.959660 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 19.430847ms to process 0 records I0416 19:17:33.959675 1 tasks_processing.go:74] worker 44 stopped. I0416 19:17:33.959683 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 18.476042ms to process 0 records I0416 19:17:33.959777 1 tasks_processing.go:74] worker 16 stopped. I0416 19:17:33.960391 1 recorder.go:75] Recording config/infrastructure with fingerprint=45457198bbe46013390c3c4a94d1b8bdd35c2945725aaa83f393db67a33ef3e8 I0416 19:17:33.960408 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 18.448458ms to process 1 records E0416 19:17:33.960420 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0416 19:17:33.960424 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 19.926539ms to process 0 records I0416 19:17:33.960431 1 tasks_processing.go:74] worker 1 stopped. I0416 19:17:33.960448 1 tasks_processing.go:74] worker 53 stopped. I0416 19:17:33.960566 1 recorder.go:75] Recording config/apiserver with fingerprint=fa5375edf3b02c2eadcf389e1e336d21e0361712045cc56e9625b21a854c5c31 I0416 19:17:33.960577 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 19.94551ms to process 1 records I0416 19:17:33.960659 1 tasks_processing.go:74] worker 63 stopped. I0416 19:17:33.960996 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=f9264bd85b3acaad7e62263845c69cd29376c59041aa3cbab0b21532908950e8 I0416 19:17:33.961007 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 18.4797ms to process 1 records I0416 19:17:33.961450 1 tasks_processing.go:74] worker 46 stopped. I0416 19:17:33.961463 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 18.927666ms to process 0 records I0416 19:17:33.962319 1 tasks_processing.go:74] worker 13 stopped. I0416 19:17:33.962471 1 recorder.go:75] Recording config/authentication with fingerprint=8ad94956e5b8d88477bbb093ae0b108819898f46970a870e3966bf214626058f I0416 19:17:33.962482 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 20.509481ms to process 1 records I0416 19:17:33.962543 1 recorder.go:75] Recording config/image with fingerprint=70deaa5d6f37d7bd9de7740341c59e46fbd1a25d7e22aa93bf747ff0f093dc30 I0416 19:17:33.962551 1 gather.go:177] gatherer "clusterconfig" function "image" took 21.957016ms to process 1 records I0416 19:17:33.962567 1 tasks_processing.go:74] worker 14 stopped. I0416 19:17:33.962591 1 recorder.go:75] Recording config/proxy with fingerprint=8b1dd3b355e4c2621d0b134cc706a6bbd44f132ff2805660ce2248dc8c95b16f I0416 19:17:33.962598 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 20.912391ms to process 1 records I0416 19:17:33.962604 1 tasks_processing.go:74] worker 45 stopped. I0416 19:17:33.962768 1 tasks_processing.go:74] worker 36 stopped. I0416 19:17:33.962849 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=f8ed02beb1313ebce36db073f9189aa814e9e9ed31ebe1e4e033e057662d404f I0416 19:17:33.962864 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 21.533628ms to process 1 records I0416 19:17:33.962875 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 22.814271ms to process 0 records I0416 19:17:33.962883 1 tasks_processing.go:74] worker 12 stopped. I0416 19:17:33.962956 1 tasks_processing.go:74] worker 37 stopped. I0416 19:17:33.963136 1 recorder.go:75] Recording config/node/ip-10-0-1-171.ec2.internal with fingerprint=57d8495e0d71cf9aadd8f226779234e2a803acee332aa31c3203a09cec185235 I0416 19:17:33.963145 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 22.846287ms to process 1 records I0416 19:17:33.963780 1 tasks_processing.go:74] worker 21 stopped. I0416 19:17:33.963794 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 22.090874ms to process 0 records I0416 19:17:33.963824 1 tasks_processing.go:74] worker 7 stopped. I0416 19:17:33.963833 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 23.77392ms to process 0 records I0416 19:17:33.964538 1 tasks_processing.go:74] worker 20 stopped. I0416 19:17:33.964550 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 23.062961ms to process 0 records I0416 19:17:33.964893 1 tasks_processing.go:74] worker 48 stopped. I0416 19:17:33.964960 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=4eaaf10331d6ab31abbe2009ebb1b534651560e65d603848570042caa20f22f6 I0416 19:17:33.964985 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 23.459272ms to process 1 records I0416 19:17:33.967901 1 tasks_processing.go:74] worker 25 stopped. I0416 19:17:33.968104 1 recorder.go:75] Recording config/featuregate with fingerprint=4f892a51c075bcb649637ba314e7ce0f2cd7a31ed916a6dc7e6d9e5f7479c0c6 I0416 19:17:33.968121 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 25.451643ms to process 1 records I0416 19:17:33.968318 1 tasks_processing.go:74] worker 60 stopped. I0416 19:17:33.968355 1 recorder.go:75] Recording config/network with fingerprint=32d8ae1897c1317fcfdddac12f6d0c8f339b02018217b523019fe26d852ba550 I0416 19:17:33.968391 1 gather.go:177] gatherer "clusterconfig" function "networks" took 26.760521ms to process 1 records I0416 19:17:33.969204 1 tasks_processing.go:74] worker 49 stopped. W0416 19:17:33.970027 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0416 19:17:33.970193 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0416 19:17:33.970221 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 29.073385ms to process 1 records I0416 19:17:33.970322 1 tasks_processing.go:74] worker 31 stopped. I0416 19:17:33.970396 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=b107f60521f6e095ab48de58455a4a9eba9de997d81ca627f66f2d360f423903 I0416 19:17:33.970436 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=9eb91e27a6de9e1b55a87aab66b223b5a21f98a799e7c104a75f3d9ae6735f8f I0416 19:17:33.970492 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 29.717445ms to process 2 records I0416 19:17:33.970831 1 gather_logs.go:145] no pods in namespace were found I0416 19:17:33.970850 1 tasks_processing.go:74] worker 8 stopped. I0416 19:17:33.970868 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 28.714149ms to process 0 records I0416 19:17:33.973714 1 tasks_processing.go:74] worker 5 stopped. I0416 19:17:33.974525 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=01910cb8a06f5f9d8ed5bc4ef40e1e9915e01ca095e83ea23dba004740d95f6d I0416 19:17:33.974628 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0416 19:17:33.974636 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0416 19:17:33.974746 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=7462cbd7599eacad0181974285956cd7333aea78f93bb26983374a68de30b322 I0416 19:17:33.974764 1 gather.go:177] gatherer "clusterconfig" function "crds" took 33.706247ms to process 2 records W0416 19:17:33.974804 1 operator.go:288] started I0416 19:17:33.974822 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0416 19:17:33.974838 1 tasks_processing.go:74] worker 11 stopped. I0416 19:17:33.974874 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=b84ce6096c7b2b8e552ea0625d31274346e38047c57d564dc821dd4344871c0c I0416 19:17:33.974895 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=c4c409f5973ec13cfc9f17fdaa5f3d5639349f4b205d74e5dd1175c2d3b29bca I0416 19:17:33.974921 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=85c41a6d34c9e09deb066a25694e44fce33b45fe23550178ea881d8b6c1796ad I0416 19:17:33.974928 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 31.538ms to process 3 records I0416 19:17:33.975003 1 tasks_processing.go:74] worker 28 stopped. I0416 19:17:33.975118 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=77153007d65871dabfa470d88b6ec51f8c9e3d5fd1b83ee8f46db51635dced22 I0416 19:17:33.975240 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=a2477f3e39c1a4f4edb49c4c0b977debcf771fbf212282dc06289cdcd46fad29 I0416 19:17:33.975253 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 32.197865ms to process 2 records I0416 19:17:33.977102 1 tasks_processing.go:74] worker 26 stopped. I0416 19:17:33.977171 1 recorder.go:75] Recording config/olm_operators with fingerprint=bd8cca38abde5e075961d7ec5eb05824413a37cb7b25940abdd9c3ea710a6fbb I0416 19:17:33.977191 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 37.070231ms to process 1 records I0416 19:17:33.981078 1 tasks_processing.go:74] worker 47 stopped. I0416 19:17:33.981141 1 recorder.go:75] Recording config/running_containers with fingerprint=cec451d1dcc91d52055e1f370c639ab202e379f4087104fc31d94ccde6aabce9 I0416 19:17:33.981153 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 39.323115ms to process 1 records I0416 19:17:33.983082 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0416 19:17:33.983093 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0416 19:17:33.983096 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0416 19:17:33.983098 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0416 19:17:33.983102 1 controller.go:212] Source scaController *sca.Controller is not ready I0416 19:17:33.983117 1 controller.go:489] The operator is still being initialized I0416 19:17:33.983122 1 controller.go:512] The operator is healthy I0416 19:17:33.984054 1 tasks_processing.go:74] worker 61 stopped. I0416 19:17:33.984068 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 42.761946ms to process 0 records I0416 19:17:33.984346 1 tasks_processing.go:74] worker 59 stopped. I0416 19:17:33.984363 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 41.750826ms to process 0 records I0416 19:17:33.987785 1 prometheus_rules.go:88] Prometheus rules successfully created E0416 19:17:33.992781 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%277fdc45f3-b8ac-4f08-bd52-807d88addcfd%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.19:56582->172.30.0.10:53: read: connection refused I0416 19:17:33.992804 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%277fdc45f3-b8ac-4f08-bd52-807d88addcfd%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.19:56582->172.30.0.10:53: read: connection refused I0416 19:17:33.994764 1 tasks_processing.go:74] worker 51 stopped. E0416 19:17:33.994779 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0416 19:17:33.994785 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0416 19:17:33.994788 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0416 19:17:33.994799 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0416 19:17:33.994826 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0416 19:17:33.994832 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0416 19:17:33.994837 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0416 19:17:33.994849 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0416 19:17:33.994887 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0416 19:17:33.994896 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0416 19:17:33.994901 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 54.687992ms to process 7 records I0416 19:17:34.009549 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0416 19:17:34.009556 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0416 19:17:34.009567 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0416 19:17:34.013488 1 base_controller.go:82] Caches are synced for ConfigController I0416 19:17:34.013505 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0416 19:17:34.017473 1 configmapobserver.go:84] configmaps "insights-config" not found I0416 19:17:34.023231 1 tasks_processing.go:74] worker 17 stopped. I0416 19:17:34.023439 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=e0145db922dbecd04eee49a1c5a6ab16cf64f7caedb08b766dfebb275cedc44c I0416 19:17:34.023522 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=6e241fb7827c2391f14ee9d4c93ba1610f77b80e963814183baf05c79f29bbab I0416 19:17:34.023541 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=c6cfb507922da6ad41f0680c5b9336c708c260faad281f74206bbb74ef587a9c I0416 19:17:34.023563 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=05e32ad5a59b90870a25ec4db79388c3875191477785746799c331b2a414e684 I0416 19:17:34.023586 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=0c3a668cd2181cf6ce4a0e1f9293dc70f01809dea6c5cfd7543aff1d00871dc2 I0416 19:17:34.023609 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=aecf5009caa471f4d80cd5b19a798f8d3d226da05500deadc76b76225cf739d5 I0416 19:17:34.023632 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=68b50057bce15c607dbc0fb95ccaa9c655ba353c30257af0aa3c341e201963c1 I0416 19:17:34.023664 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=3ca4592b41ea3f009d009ef486f107a69578a0cbad30a1466b235d93eb1d0b65 I0416 19:17:34.023687 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=e4536573139b088eeffbf290b407f108d15622ff6e3b0d6913d0153647b098da I0416 19:17:34.023718 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=c99fc2a96ceaffaa9b58715f84fb74062313451a74aedb00cf0c94165eff481b I0416 19:17:34.023741 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=dd7b64cf2a817ee259edca188e79c6d775d1cff8b3d4f0847914829d2ba29c41 I0416 19:17:34.023750 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 83.226607ms to process 11 records I0416 19:17:34.026025 1 tasks_processing.go:74] worker 42 stopped. I0416 19:17:34.026149 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=9ccff5c47c8725916ab8ad9c1dc72681d71a00064f8e284e401370de690642f9 I0416 19:17:34.026185 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=f07fa9f016c928f8d853a80e31404ee0293d2f76138e05115cf174277019553f I0416 19:17:34.026211 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=175c7713475c3d25f51959e6c10cad299aa667cd8d1840eacaac5de52fe4b255 I0416 19:17:34.026219 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 86.044275ms to process 3 records I0416 19:17:34.062477 1 tasks_processing.go:74] worker 0 stopped. I0416 19:17:34.062787 1 recorder.go:75] Recording config/version with fingerprint=d7b758e95be1838183e5c3385c5216b38e2e59ec05bf067a8121e90ffbd6d30a I0416 19:17:34.062803 1 recorder.go:75] Recording config/id with fingerprint=af2861fdb94df0f6ce0bc0e5770982f0bad444a446200cb7803c0cc40ab9ca72 I0416 19:17:34.062809 1 gather.go:177] gatherer "clusterconfig" function "version" took 122.706756ms to process 2 records I0416 19:17:34.065268 1 tasks_processing.go:74] worker 52 stopped. I0416 19:17:34.065309 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0416 19:17:34.065319 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 123.345361ms to process 1 records I0416 19:17:34.071289 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0416 19:17:34.074967 1 base_controller.go:82] Caches are synced for LoggingSyncer W0416 19:17:34.074974 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.19:51904->172.30.0.10:53: read: connection refused I0416 19:17:34.074982 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0416 19:17:34.074986 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.19:51904->172.30.0.10:53: read: connection refused I0416 19:17:34.082326 1 tasks_processing.go:74] worker 24 stopped. E0416 19:17:34.082341 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0416 19:17:34.082347 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2pne4p9skicgtffsqp0c4hmdek8jouie-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2pne4p9skicgtffsqp0c4hmdek8jouie-primary-cert-bundle-secret" not found I0416 19:17:34.082418 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=0ccfe4cb9f14fb2ac334df4b5bf000e531fe4a20427f6e83b777adb472fc9d53 I0416 19:17:34.082432 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 141.341683ms to process 1 records I0416 19:17:34.378826 1 gather_cluster_operator_pods_and_events.go:121] Found 10 pods with 11 containers I0416 19:17:34.378844 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 2287802 bytes I0416 19:17:34.378930 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-n6fx7 pod in namespace openshift-dns (previous: false). I0416 19:17:34.603499 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-n6fx7 pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-n6fx7\" is waiting to start: ContainerCreating" I0416 19:17:34.603520 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-n6fx7\" is waiting to start: ContainerCreating" I0416 19:17:34.603528 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-n6fx7 pod in namespace openshift-dns (previous: false). I0416 19:17:34.785044 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-n6fx7 pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-n6fx7\" is waiting to start: ContainerCreating" I0416 19:17:34.785068 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-n6fx7\" is waiting to start: ContainerCreating" I0416 19:17:34.785081 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-d67dw pod in namespace openshift-dns (previous: false). W0416 19:17:34.969055 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0416 19:17:34.984118 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0416 19:17:34.984172 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-57f764466d-dmhm5 pod in namespace openshift-image-registry (previous: false). I0416 19:17:35.180137 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0416 19:17:35.180210 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-57f764466d-l4tfr pod in namespace openshift-image-registry (previous: false). I0416 19:17:35.383668 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-57f764466d-l4tfr pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-57f764466d-l4tfr\" is waiting to start: ContainerCreating" I0416 19:17:35.383686 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-57f764466d-l4tfr\" is waiting to start: ContainerCreating" I0416 19:17:35.383736 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-5f8d98cb75-lb7mq pod in namespace openshift-image-registry (previous: false). I0416 19:17:35.388386 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0416 19:17:35.590491 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0416 19:17:35.590513 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-z6m78 pod in namespace openshift-image-registry (previous: false). I0416 19:17:35.784890 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0416 19:17:35.784912 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-64f7ffd785-rmrc6 pod in namespace openshift-ingress (previous: false). I0416 19:17:35.791586 1 tasks_processing.go:74] worker 56 stopped. I0416 19:17:35.791639 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=8e43eef7a7586ce12cce372e82fa1fc2cbb3ded1653a8c3e5ef9e314e80dad6d I0416 19:17:35.791668 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=53d367b2972e00f3d1f7dff3f6cf4d320839248e470e1d0ea84fd28e9e641529 I0416 19:17:35.791699 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0416 19:17:35.791728 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=2c09805b5f138008f3099777761badbb94ba2dd6f67c0da2a642c7a3c5e88286 I0416 19:17:35.791744 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0416 19:17:35.791769 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=6754112a5c7cd9ba546780d060a28e0a910f6d35b6ae762648f1614a54320da0 I0416 19:17:35.791799 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=73612b38a4727562d0bc70ad7fabdde5b24a2a912a16389e566d6d6ac1d1b915 I0416 19:17:35.791837 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=b16d356c4f85d91efc8e75f32635880203a8075390ba730521633c70d3e87cc0 I0416 19:17:35.791852 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=7fe144924d95bd7b9675d6cfaa4cc9432b8c5daffcc86c1bee676335019dc7b2 I0416 19:17:35.791872 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=8fb0405313333a467a0a6178fcb6aca314fae0121e0ec3ef39289f293fbd7e59 I0416 19:17:35.791880 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0416 19:17:35.791895 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=2dee0b3f12068d86e5ef3044f45ae1ba02ea626afbd63ac4c635154569b91110 I0416 19:17:35.791905 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0416 19:17:35.791921 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=e872486dbf3fd9a1a4e7abc8615cba8a236ac7a15c713cbdda8321d456a1b74d I0416 19:17:35.791930 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0416 19:17:35.791945 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=cb6dbe6b7903e207f6a7e5d1ebb072b39b35e2911fe3473fcc8ab42439d1c707 I0416 19:17:35.791955 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0416 19:17:35.791971 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=7b38c1e3264119ef9401033000a7c7c4cbacf99e5b3a250e36df2664c9aeb99d I0416 19:17:35.792090 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=3fd601fa461ffe925227495accc4efccf4462db472ae6d891c250ee9e6472122 I0416 19:17:35.792099 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0416 19:17:35.792106 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0416 19:17:35.792125 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0416 19:17:35.792148 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=c2dc1bbdcb7566ed9fc9cd6cb4dba1e9d551d093f1a91c4ac054b5a18b255c42 I0416 19:17:35.792172 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=a481a54788861445f53b36cc8367e7cb5cdc5cc9b46579787a1c6bbb9c50b4ef I0416 19:17:35.792180 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0416 19:17:35.792197 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=59509e0d62668e8f2c2098f92bc2e26cfcf51e65f5ef0bcbbe3ce68374e03a35 I0416 19:17:35.792207 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0416 19:17:35.792219 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=01fde2fbf3f87782ca0b6ff2f08441085b321e98bce14f938d4c1538bf012ffe I0416 19:17:35.792233 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=82d300b94c94955b58e91ec3009e7399398e912dad1329f1bd3c3d7dbf11730e I0416 19:17:35.792249 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=7cc109f09fe4330604e2b34a4ab11e5e956f5ff4a5da37cb9543c407fa84945f I0416 19:17:35.792264 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=e3b539872d9aca553438dea730a1e3541354d754037f772997da4ee095df339d I0416 19:17:35.792277 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=6ae7347b0dfa8338bc0a80a69e6a09563893e82492183306fe996427fe7831e9 I0416 19:17:35.792315 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=91838c1f6d9a34b792d0be5dd5dbe247f1583ee9a3f537883955eddd062b3eb5 I0416 19:17:35.792331 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0416 19:17:35.792338 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0416 19:17:35.792344 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.851589083s to process 35 records W0416 19:17:35.968723 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0416 19:17:35.988222 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-64f7ffd785-rmrc6 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-64f7ffd785-rmrc6\" is waiting to start: ContainerCreating" I0416 19:17:35.988238 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-64f7ffd785-rmrc6\" is waiting to start: ContainerCreating" I0416 19:17:35.988249 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-b59cd5948-4scmx pod in namespace openshift-ingress (previous: false). I0416 19:17:36.184427 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-b59cd5948-4scmx pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-b59cd5948-4scmx\" is waiting to start: ContainerCreating" I0416 19:17:36.184446 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-b59cd5948-4scmx\" is waiting to start: ContainerCreating" I0416 19:17:36.184459 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-b59cd5948-l6cbg pod in namespace openshift-ingress (previous: false). I0416 19:17:36.378467 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0416 19:17:36.378517 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-w62vb pod in namespace openshift-ingress-canary (previous: false). I0416 19:17:36.586979 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-w62vb pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-w62vb\" is waiting to start: ContainerCreating" I0416 19:17:36.586996 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-w62vb\" is waiting to start: ContainerCreating" I0416 19:17:36.587015 1 tasks_processing.go:74] worker 22 stopped. I0416 19:17:36.587083 1 recorder.go:75] Recording events/openshift-dns with fingerprint=85e1dc2912b248e56dbb69b8c7f77c62347f4bbc64eefa2872faa3c2591d7ea5 I0416 19:17:36.587206 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=af5b800a961e2982529371b198e957cb68936482306f7e3bfff110a5f729ee20 I0416 19:17:36.587252 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=81a7b597d8944ea2c87d16b86f689c1a0feb147fad08ba807abe793a4b6caea3 I0416 19:17:36.587293 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=0d2b941d7566485d1e61e94f8ec72cb688584f797f36103f30543cb4e920fcb1 I0416 19:17:36.587322 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=2db36b7bce1a8ed06c87be0dde4bf3f01a1dd813d39ddeee7180d7f1267f3f5c I0416 19:17:36.588600 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-n6fx7 with fingerprint=8131616a1878c0b81aa13286d341412298ae573796845c803b217ee052eb368a I0416 19:17:36.588700 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-57f764466d-dmhm5 with fingerprint=f4cd811ec97ba1fec708b848f439927ac7eb439531b21f047e5039558c910b0a I0416 19:17:36.588797 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-57f764466d-l4tfr with fingerprint=91fc44201676c4450f61676484947e6e69ea3182364eabded08873dcf0ab4b4b I0416 19:17:36.588872 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-5f8d98cb75-lb7mq with fingerprint=6eed7f594d3e106d1f97e27d75fd3ba43a536e45cd727cbc66c44df87d8e4e15 I0416 19:17:36.588933 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-w62vb with fingerprint=b95f92e595c5cbed243dbade426d7da69433e86cdc3d7af56e01c0ecafae3f86 I0416 19:17:36.588942 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 2.646480899s to process 10 records W0416 19:17:36.968896 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0416 19:17:37.968971 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0416 19:17:38.965180 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0416 19:17:38.965206 1 tasks_processing.go:74] worker 18 stopped. E0416 19:17:38.965218 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0416 19:17:38.965229 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0416 19:17:38.965257 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0416 19:17:38.965266 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.024078863s to process 1 records I0416 19:17:42.990279 1 configmapobserver.go:84] configmaps "insights-config" not found I0416 19:17:46.380928 1 tasks_processing.go:74] worker 58 stopped. I0416 19:17:46.380984 1 recorder.go:75] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0416 19:17:46.380998 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.440486887s to process 1 records I0416 19:17:47.149656 1 tasks_processing.go:74] worker 33 stopped. I0416 19:17:47.149991 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=751aa8b7d1208af5fd734152a53c8f2354708612b0cd50cfd9c595651a5cc8c8 I0416 19:17:47.150007 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.209662863s to process 1 records E0416 19:17:47.150067 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.21s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machines\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0416 19:17:47.151168 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machines" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0416 19:17:47.151183 1 periodic.go:209] Running workloads gatherer I0416 19:17:47.151198 1 tasks_processing.go:45] number of workers: 2 I0416 19:17:47.151204 1 tasks_processing.go:69] worker 1 listening for tasks. I0416 19:17:47.151208 1 tasks_processing.go:71] worker 1 working on workload_info task. I0416 19:17:47.151215 1 tasks_processing.go:69] worker 0 listening for tasks. I0416 19:17:47.151237 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0416 19:17:47.168548 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 19s for image data I0416 19:17:47.174362 1 tasks_processing.go:74] worker 0 stopped. I0416 19:17:47.174377 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 23.105477ms to process 0 records I0416 19:17:47.177155 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (9ms) I0416 19:17:47.185357 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (8ms) I0416 19:17:47.193593 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (8ms) I0416 19:17:47.205449 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (12ms) I0416 19:17:47.213885 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (8ms) I0416 19:17:47.221626 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (8ms) I0416 19:17:47.229665 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (8ms) I0416 19:17:47.238976 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (9ms) I0416 19:17:47.247202 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (8ms) I0416 19:17:47.255481 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (8ms) I0416 19:17:47.277658 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (22ms) I0416 19:17:47.379641 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (102ms) I0416 19:17:47.478225 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (99ms) I0416 19:17:47.577855 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (100ms) I0416 19:17:47.678433 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (101ms) I0416 19:17:47.778006 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (100ms) I0416 19:17:47.879203 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (101ms) I0416 19:17:47.978199 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (99ms) I0416 19:17:48.077780 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (100ms) I0416 19:17:48.178482 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (101ms) I0416 19:17:48.207100 1 configmapobserver.go:84] configmaps "insights-config" not found I0416 19:17:48.279343 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (101ms) I0416 19:17:48.378481 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (99ms) I0416 19:17:48.404826 1 configmapobserver.go:84] configmaps "insights-config" not found I0416 19:17:48.478938 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (100ms) I0416 19:17:48.578544 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0416 19:17:48.679751 1 gather_workloads_info.go:387] No image sha256:bffebf689df67af02a06b97d3d9bfcfdccd597350d132171ee68f3d0ed29a3f6 (101ms) I0416 19:17:48.778717 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (99ms) I0416 19:17:49.084215 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (305ms) I0416 19:17:49.092657 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (8ms) I0416 19:17:49.100810 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (8ms) I0416 19:17:49.177496 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (77ms) I0416 19:17:49.277878 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (100ms) I0416 19:17:49.378212 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (100ms) I0416 19:17:49.478159 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (100ms) I0416 19:17:49.478192 1 tasks_processing.go:74] worker 1 stopped. E0416 19:17:49.478203 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0416 19:17:49.478455 1 recorder.go:75] Recording config/workload_info with fingerprint=aba46471c6552a619f03e6127c04f1d7d82307600ba061cde5dce718d9ce96f8 I0416 19:17:49.478470 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.326976054s to process 1 records E0416 19:17:49.478506 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.327s with: function \"workload_info\" failed with an error" I0416 19:17:49.479609 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0416 19:17:49.479622 1 periodic.go:209] Running conditional gatherer I0416 19:17:49.484724 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0416 19:17:49.492356 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.19:57394->172.30.0.10:53: read: connection refused E0416 19:17:49.492623 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0416 19:17:49.492684 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0416 19:17:49.497837 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0416 19:17:49.497855 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0416 19:17:49.497863 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0416 19:17:49.497868 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0416 19:17:49.497871 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0416 19:17:49.497875 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0416 19:17:49.497880 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0416 19:17:49.497884 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0416 19:17:49.497889 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0416 19:17:49.497893 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0416 19:17:49.497914 1 tasks_processing.go:45] number of workers: 3 I0416 19:17:49.497937 1 tasks_processing.go:69] worker 2 listening for tasks. I0416 19:17:49.497944 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0416 19:17:49.497946 1 tasks_processing.go:69] worker 0 listening for tasks. I0416 19:17:49.497959 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0416 19:17:49.497959 1 tasks_processing.go:69] worker 1 listening for tasks. I0416 19:17:49.497971 1 tasks_processing.go:74] worker 1 stopped. I0416 19:17:49.497971 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0416 19:17:49.498030 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0416 19:17:49.498046 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.111µs to process 1 records I0416 19:17:49.498113 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0416 19:17:49.498124 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.072µs to process 1 records I0416 19:17:49.498136 1 tasks_processing.go:74] worker 2 stopped. I0416 19:17:49.498209 1 tasks_processing.go:74] worker 0 stopped. I0416 19:17:49.498223 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 240.535µs to process 0 records I0416 19:17:49.498243 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.19:57394->172.30.0.10:53: read: connection refused I0416 19:17:49.498260 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0416 19:17:49.518852 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=1c2d15e11c2711fe51c5285afe5bd3122bb1fb9064b09c42491c3d089e7fccb8 I0416 19:17:49.519026 1 diskrecorder.go:70] Writing 102 records to /var/lib/insights-operator/insights-2026-04-16-191749.tar.gz I0416 19:17:49.525521 1 diskrecorder.go:51] Wrote 102 records to disk in 6ms I0416 19:17:49.525559 1 periodic.go:278] Gathering cluster info every 2h0m0s I0416 19:17:49.525578 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0416 19:17:59.333090 1 configmapobserver.go:84] configmaps "insights-config" not found I0416 19:18:43.400992 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="064f63566b35eb4251c7ce9a24d879bbe09e84abeb9593e6321eeeeebdf8be02") W0416 19:18:43.401026 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.key was created I0416 19:18:43.401090 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0416 19:18:43.401113 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0416 19:18:43.401114 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="4c35c632acc08e2af86d6ce85aacd7a9b7f5accfc7c27146952d30fc08c19172") I0416 19:18:43.401164 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0416 19:18:43.401172 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="021c15d7a3237239a18b70810f154b28a8a3eaca0c4bb36fefa4b2bb1a8128c7") I0416 19:18:43.401188 1 requestheader_controller.go:194] Shutting down RequestHeaderAuthRequestController I0416 19:18:43.401188 1 base_controller.go:181] Shutting down ConfigController ... I0416 19:18:43.401203 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0416 19:18:43.401214 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0416 19:18:43.401223 1 base_controller.go:113] All ConfigController workers have been terminated I0416 19:18:43.401222 1 periodic.go:170] Shutting down I0416 19:18:43.401203 1 base_controller.go:181] Shutting down LoggingSyncer ... I0416 19:18:43.401212 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0416 19:18:43.401232 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0416 19:18:43.401246 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0416 19:18:43.401166 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0416 19:18:43.401249 1 tlsconfig.go:258] "Shutting down DynamicServingCertificateController" I0416 19:18:43.401188 1 genericapiserver.go:651] "[graceful-termination] not going to wait for active watch request(s) to drain" I0416 19:18:43.401252 1 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/tmp/serving-cert-1058509088/tls.crt::/tmp/serving-cert-1058509088/tls.key"