W0504 20:42:10.257033 1 cmd.go:257] Using insecure, self-signed certificates I0504 20:42:11.658389 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:42:11.658846 1 observer_polling.go:159] Starting file observer I0504 20:42:12.570753 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0504 20:42:12.570944 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0504 20:42:12.571528 1 secure_serving.go:57] Forcing use of http/1.1 only I0504 20:42:12.571535 1 simple_featuregate_reader.go:171] Starting feature-gate-detector W0504 20:42:12.571553 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0504 20:42:12.571558 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0504 20:42:12.571565 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0504 20:42:12.571569 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0504 20:42:12.571573 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0504 20:42:12.571578 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0504 20:42:12.575146 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0504 20:42:12.575209 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"d2e69c47-e69c-41c6-b52a-573c7fe2937e", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0504 20:42:12.576737 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0504 20:42:12.576741 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0504 20:42:12.576745 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0504 20:42:12.576754 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0504 20:42:12.576761 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0504 20:42:12.576755 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0504 20:42:12.577018 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-2990929576/tls.crt::/tmp/serving-cert-2990929576/tls.key" I0504 20:42:12.577227 1 secure_serving.go:213] Serving securely on [::]:8443 I0504 20:42:12.577256 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0504 20:42:12.580460 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0504 20:42:12.580482 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0504 20:42:12.580540 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0504 20:42:12.584991 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 20:42:12.585008 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0504 20:42:12.590545 1 secretconfigobserver.go:119] support secret does not exist I0504 20:42:12.595130 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 20:42:12.599444 1 secretconfigobserver.go:119] support secret does not exist I0504 20:42:12.602164 1 recorder.go:161] Pruning old reports every 5h5m5s, max age is 288h0m0s I0504 20:42:12.607543 1 periodic.go:209] Running clusterconfig gatherer I0504 20:42:12.607546 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0504 20:42:12.607553 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0504 20:42:12.607565 1 insightsreport.go:296] Starting report retriever I0504 20:42:12.607563 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0504 20:42:12.607570 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0504 20:42:12.607578 1 tasks_processing.go:45] number of workers: 64 I0504 20:42:12.607600 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 20:42:12.607613 1 tasks_processing.go:69] worker 4 listening for tasks. I0504 20:42:12.607604 1 tasks_processing.go:69] worker 6 listening for tasks. I0504 20:42:12.607608 1 tasks_processing.go:69] worker 14 listening for tasks. I0504 20:42:12.607622 1 tasks_processing.go:71] worker 4 working on lokistack task. I0504 20:42:12.607626 1 tasks_processing.go:69] worker 15 listening for tasks. I0504 20:42:12.607628 1 tasks_processing.go:69] worker 12 listening for tasks. I0504 20:42:12.607631 1 tasks_processing.go:69] worker 39 listening for tasks. I0504 20:42:12.607632 1 tasks_processing.go:69] worker 2 listening for tasks. I0504 20:42:12.607620 1 tasks_processing.go:71] worker 0 working on machine_healthchecks task. I0504 20:42:12.607642 1 tasks_processing.go:69] worker 13 listening for tasks. I0504 20:42:12.607643 1 tasks_processing.go:69] worker 41 listening for tasks. I0504 20:42:12.607638 1 tasks_processing.go:69] worker 10 listening for tasks. I0504 20:42:12.607652 1 tasks_processing.go:69] worker 46 listening for tasks. I0504 20:42:12.607653 1 tasks_processing.go:69] worker 8 listening for tasks. I0504 20:42:12.607659 1 tasks_processing.go:69] worker 53 listening for tasks. I0504 20:42:12.607661 1 tasks_processing.go:69] worker 42 listening for tasks. I0504 20:42:12.607643 1 tasks_processing.go:69] worker 29 listening for tasks. I0504 20:42:12.607664 1 tasks_processing.go:69] worker 16 listening for tasks. I0504 20:42:12.607623 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 20:42:12.607620 1 tasks_processing.go:69] worker 11 listening for tasks. I0504 20:42:12.607676 1 tasks_processing.go:69] worker 44 listening for tasks. I0504 20:42:12.607635 1 tasks_processing.go:69] worker 40 listening for tasks. I0504 20:42:12.607683 1 tasks_processing.go:69] worker 37 listening for tasks. I0504 20:42:12.607679 1 tasks_processing.go:69] worker 9 listening for tasks. I0504 20:42:12.607686 1 tasks_processing.go:69] worker 45 listening for tasks. I0504 20:42:12.607690 1 tasks_processing.go:69] worker 38 listening for tasks. I0504 20:42:12.607689 1 tasks_processing.go:69] worker 57 listening for tasks. I0504 20:42:12.607693 1 tasks_processing.go:69] worker 19 listening for tasks. I0504 20:42:12.607697 1 tasks_processing.go:69] worker 62 listening for tasks. I0504 20:42:12.607639 1 tasks_processing.go:69] worker 28 listening for tasks. I0504 20:42:12.607694 1 tasks_processing.go:69] worker 51 listening for tasks. I0504 20:42:12.607699 1 tasks_processing.go:69] worker 58 listening for tasks. I0504 20:42:12.607707 1 tasks_processing.go:69] worker 63 listening for tasks. I0504 20:42:12.607704 1 tasks_processing.go:69] worker 61 listening for tasks. I0504 20:42:12.607707 1 tasks_processing.go:69] worker 60 listening for tasks. I0504 20:42:12.607710 1 tasks_processing.go:69] worker 59 listening for tasks. I0504 20:42:12.607709 1 tasks_processing.go:69] worker 23 listening for tasks. I0504 20:42:12.607715 1 tasks_processing.go:69] worker 22 listening for tasks. I0504 20:42:12.607717 1 tasks_processing.go:69] worker 26 listening for tasks. I0504 20:42:12.607724 1 tasks_processing.go:71] worker 41 working on storage_classes task. I0504 20:42:12.607726 1 tasks_processing.go:71] worker 13 working on jaegers task. I0504 20:42:12.607727 1 tasks_processing.go:71] worker 12 working on overlapping_namespace_uids task. I0504 20:42:12.607650 1 tasks_processing.go:69] worker 30 listening for tasks. I0504 20:42:12.607727 1 tasks_processing.go:71] worker 46 working on operators_pods_and_events task. I0504 20:42:12.607711 1 tasks_processing.go:69] worker 49 listening for tasks. I0504 20:42:12.607733 1 tasks_processing.go:71] worker 2 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0504 20:42:12.607749 1 tasks_processing.go:71] worker 37 working on machine_configs task. I0504 20:42:12.607756 1 tasks_processing.go:71] worker 49 working on infrastructures task. I0504 20:42:12.607764 1 tasks_processing.go:71] worker 58 working on machine_config_pools task. I0504 20:42:12.607615 1 tasks_processing.go:69] worker 3 listening for tasks. I0504 20:42:12.607806 1 tasks_processing.go:71] worker 57 working on feature_gates task. I0504 20:42:12.607813 1 tasks_processing.go:71] worker 3 working on image_registries task. I0504 20:42:12.607815 1 tasks_processing.go:71] worker 9 working on config_maps task. I0504 20:42:12.607825 1 tasks_processing.go:71] worker 38 working on version task. I0504 20:42:12.607810 1 tasks_processing.go:71] worker 45 working on ingress task. I0504 20:42:12.607904 1 tasks_processing.go:71] worker 42 working on networks task. I0504 20:42:12.607911 1 tasks_processing.go:71] worker 28 working on openstack_controlplanes task. I0504 20:42:12.607925 1 tasks_processing.go:71] worker 23 working on authentication task. I0504 20:42:12.607970 1 tasks_processing.go:71] worker 59 working on machine_sets task. I0504 20:42:12.608032 1 tasks_processing.go:71] worker 26 working on sap_datahubs task. I0504 20:42:12.608089 1 tasks_processing.go:71] worker 30 working on machines task. I0504 20:42:12.608090 1 tasks_processing.go:71] worker 51 working on tsdb_status task. I0504 20:42:12.608104 1 tasks_processing.go:71] worker 63 working on machine_autoscalers task. W0504 20:42:12.608128 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:42:12.608157 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 38.326µs to process 0 records I0504 20:42:12.608168 1 tasks_processing.go:71] worker 60 working on cluster_apiserver task. I0504 20:42:12.607664 1 tasks_processing.go:69] worker 33 listening for tasks. I0504 20:42:12.607668 1 tasks_processing.go:69] worker 54 listening for tasks. I0504 20:42:12.607669 1 tasks_processing.go:69] worker 43 listening for tasks. I0504 20:42:12.607670 1 tasks_processing.go:69] worker 34 listening for tasks. I0504 20:42:12.607664 1 tasks_processing.go:69] worker 52 listening for tasks. I0504 20:42:12.607674 1 tasks_processing.go:69] worker 35 listening for tasks. I0504 20:42:12.607675 1 tasks_processing.go:69] worker 55 listening for tasks. I0504 20:42:12.607678 1 tasks_processing.go:69] worker 36 listening for tasks. I0504 20:42:12.607678 1 tasks_processing.go:69] worker 17 listening for tasks. I0504 20:42:12.607678 1 tasks_processing.go:69] worker 47 listening for tasks. I0504 20:42:12.607682 1 tasks_processing.go:69] worker 56 listening for tasks. I0504 20:42:12.607686 1 tasks_processing.go:69] worker 18 listening for tasks. I0504 20:42:12.607686 1 tasks_processing.go:69] worker 50 listening for tasks. I0504 20:42:12.607697 1 tasks_processing.go:69] worker 48 listening for tasks. I0504 20:42:12.607703 1 tasks_processing.go:69] worker 20 listening for tasks. I0504 20:42:12.607710 1 tasks_processing.go:69] worker 24 listening for tasks. I0504 20:42:12.607710 1 tasks_processing.go:69] worker 5 listening for tasks. I0504 20:42:12.607713 1 tasks_processing.go:69] worker 21 listening for tasks. I0504 20:42:12.607720 1 tasks_processing.go:71] worker 14 working on nodes task. I0504 20:42:12.608255 1 tasks_processing.go:71] worker 22 working on openstack_dataplanenodesets task. I0504 20:42:12.608368 1 tasks_processing.go:71] worker 19 working on image_pruners task. I0504 20:42:12.608413 1 tasks_processing.go:71] worker 1 working on proxies task. I0504 20:42:12.608486 1 tasks_processing.go:71] worker 29 working on service_accounts task. I0504 20:42:12.607660 1 tasks_processing.go:69] worker 32 listening for tasks. I0504 20:42:12.607722 1 tasks_processing.go:71] worker 39 working on openshift_machine_api_events task. I0504 20:42:12.608568 1 tasks_processing.go:71] worker 44 working on node_logs task. I0504 20:42:12.608642 1 tasks_processing.go:71] worker 62 working on pod_network_connectivity_checks task. I0504 20:42:12.608666 1 tasks_processing.go:71] worker 11 working on support_secret task. I0504 20:42:12.608705 1 tasks_processing.go:71] worker 40 working on openstack_dataplanedeployments task. I0504 20:42:12.607652 1 tasks_processing.go:69] worker 27 listening for tasks. I0504 20:42:12.607725 1 tasks_processing.go:71] worker 10 working on openshift_logging task. I0504 20:42:12.608855 1 tasks_processing.go:71] worker 27 working on sap_pods task. I0504 20:42:12.608929 1 tasks_processing.go:71] worker 16 working on container_images task. I0504 20:42:12.607727 1 tasks_processing.go:71] worker 15 working on monitoring_persistent_volumes task. I0504 20:42:12.609000 1 tasks_processing.go:71] worker 17 working on crds task. I0504 20:42:12.609079 1 tasks_processing.go:71] worker 48 working on metrics task. I0504 20:42:12.609090 1 tasks_processing.go:71] worker 51 working on aggregated_monitoring_cr_names task. I0504 20:42:12.609099 1 tasks_processing.go:71] worker 56 working on mutating_webhook_configurations task. I0504 20:42:12.607655 1 tasks_processing.go:69] worker 31 listening for tasks. W0504 20:42:12.609105 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:42:12.609115 1 tasks_processing.go:71] worker 31 working on certificate_signing_requests task. I0504 20:42:12.609122 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 25.071µs to process 0 records I0504 20:42:12.609133 1 tasks_processing.go:71] worker 47 working on image task. I0504 20:42:12.609141 1 tasks_processing.go:71] worker 50 working on schedulers task. I0504 20:42:12.609167 1 tasks_processing.go:71] worker 34 working on dvo_metrics task. I0504 20:42:12.609214 1 tasks_processing.go:71] worker 21 working on clusterroles task. I0504 20:42:12.609133 1 tasks_processing.go:71] worker 18 working on active_alerts task. W0504 20:42:12.609323 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:42:12.609337 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 23.536µs to process 0 records I0504 20:42:12.608099 1 tasks_processing.go:71] worker 61 working on oauths task. I0504 20:42:12.609182 1 tasks_processing.go:71] worker 54 working on container_runtime_configs task. I0504 20:42:12.607724 1 tasks_processing.go:71] worker 6 working on ingress_certificates task. I0504 20:42:12.609186 1 tasks_processing.go:71] worker 33 working on sap_config task. I0504 20:42:12.609448 1 tasks_processing.go:71] worker 18 working on nodenetworkconfigurationpolicies task. I0504 20:42:12.609448 1 tasks_processing.go:71] worker 48 working on install_plans task. I0504 20:42:12.609194 1 tasks_processing.go:71] worker 52 working on openstack_version task. I0504 20:42:12.607716 1 tasks_processing.go:69] worker 25 listening for tasks. I0504 20:42:12.609515 1 tasks_processing.go:74] worker 25 stopped. I0504 20:42:12.609198 1 tasks_processing.go:71] worker 43 working on operators task. I0504 20:42:12.609190 1 tasks_processing.go:71] worker 35 working on olm_operators task. I0504 20:42:12.609209 1 tasks_processing.go:71] worker 55 working on pdbs task. I0504 20:42:12.609210 1 tasks_processing.go:71] worker 24 working on silenced_alerts task. W0504 20:42:12.610065 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:42:12.609213 1 tasks_processing.go:71] worker 36 working on ceph_cluster task. I0504 20:42:12.610093 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 115.739µs to process 0 records I0504 20:42:12.609221 1 tasks_processing.go:71] worker 32 working on qemu_kubevirt_launcher_logs task. I0504 20:42:12.607614 1 tasks_processing.go:69] worker 7 listening for tasks. I0504 20:42:12.607900 1 tasks_processing.go:71] worker 53 working on validating_webhook_configurations task. I0504 20:42:12.610298 1 tasks_processing.go:74] worker 7 stopped. I0504 20:42:12.609207 1 tasks_processing.go:71] worker 20 working on cost_management_metrics_configs task. I0504 20:42:12.609203 1 tasks_processing.go:71] worker 5 working on storage_cluster task. I0504 20:42:12.607895 1 tasks_processing.go:71] worker 8 working on nodenetworkstates task. I0504 20:42:12.610084 1 tasks_processing.go:74] worker 24 stopped. I0504 20:42:12.611176 1 tasks_processing.go:74] worker 4 stopped. I0504 20:42:12.611198 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 3.542961ms to process 0 records I0504 20:42:12.611739 1 tasks_processing.go:74] worker 0 stopped. E0504 20:42:12.611755 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0504 20:42:12.611768 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 4.091716ms to process 0 records I0504 20:42:12.612587 1 tasks_processing.go:74] worker 13 stopped. I0504 20:42:12.612599 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 4.850057ms to process 0 records I0504 20:42:12.612605 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 4.668494ms to process 0 records I0504 20:42:12.612610 1 tasks_processing.go:74] worker 28 stopped. I0504 20:42:12.613515 1 tasks_processing.go:74] worker 22 stopped. I0504 20:42:12.613529 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 5.217303ms to process 0 records I0504 20:42:12.617373 1 tasks_processing.go:74] worker 26 stopped. I0504 20:42:12.617383 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 9.284011ms to process 0 records I0504 20:42:12.617395 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 8.657603ms to process 0 records I0504 20:42:12.617403 1 tasks_processing.go:74] worker 10 stopped. I0504 20:42:12.617546 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0504 20:42:12.617563 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0504 20:42:12.617568 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0504 20:42:12.617571 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0504 20:42:12.617585 1 controller.go:489] The operator is still being initialized I0504 20:42:12.617592 1 controller.go:512] The operator is healthy I0504 20:42:12.626272 1 tasks_processing.go:74] worker 45 stopped. I0504 20:42:12.626473 1 recorder.go:75] Recording config/ingress with fingerprint=c9de654b5406ea5df68bfc62e591067bbf2bc88529a32f5ea2c41dafd2344d63 I0504 20:42:12.626491 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 18.36415ms to process 1 records I0504 20:42:12.626636 1 tasks_processing.go:74] worker 23 stopped. I0504 20:42:12.626693 1 recorder.go:75] Recording config/authentication with fingerprint=fb254a956c4ea477fb26d7270a1b40605348c2aa69025498297edbf5be821f5a I0504 20:42:12.626705 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 18.360741ms to process 1 records I0504 20:42:12.626710 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 16.556428ms to process 0 records I0504 20:42:12.626715 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 17.787316ms to process 0 records I0504 20:42:12.626719 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 16.284637ms to process 0 records I0504 20:42:12.626722 1 tasks_processing.go:74] worker 36 stopped. I0504 20:42:12.626726 1 tasks_processing.go:74] worker 27 stopped. I0504 20:42:12.626731 1 tasks_processing.go:74] worker 20 stopped. I0504 20:42:12.626800 1 tasks_processing.go:74] worker 42 stopped. I0504 20:42:12.626847 1 recorder.go:75] Recording config/network with fingerprint=2637a38f0a11922eb9aee1b65d65df3271e1c20d2a6f3c87cd2fc397b2e63772 I0504 20:42:12.626859 1 gather.go:177] gatherer "clusterconfig" function "networks" took 18.766656ms to process 1 records E0504 20:42:12.626866 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0504 20:42:12.626876 1 gather.go:177] gatherer "clusterconfig" function "machines" took 18.615544ms to process 0 records I0504 20:42:12.626888 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 18.062618ms to process 0 records I0504 20:42:12.626910 1 tasks_processing.go:74] worker 30 stopped. I0504 20:42:12.626923 1 tasks_processing.go:74] worker 40 stopped. I0504 20:42:12.626992 1 tasks_processing.go:74] worker 49 stopped. I0504 20:42:12.627445 1 recorder.go:75] Recording config/infrastructure with fingerprint=923b87415cfce45731270ce263732c0ad36a8ef31a1024b12b19b3c2adf701ae I0504 20:42:12.627459 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 19.10235ms to process 1 records I0504 20:42:12.627518 1 tasks_processing.go:74] worker 19 stopped. I0504 20:42:12.627646 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=171e4366cca15991c381e781523908e01beb588b04512df8431ebf80661eab6a I0504 20:42:12.627656 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 18.682615ms to process 1 records E0504 20:42:12.627665 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0504 20:42:12.627671 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 18.409577ms to process 0 records I0504 20:42:12.627735 1 tasks_processing.go:74] worker 62 stopped. I0504 20:42:12.627804 1 tasks_processing.go:74] worker 3 stopped. I0504 20:42:12.627933 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=1b91fd886e57793fb26eeb2141b3c6bbf83698413d5b48828ea3461841ddc9a8 I0504 20:42:12.627943 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 19.543687ms to process 1 records I0504 20:42:12.627973 1 tasks_processing.go:74] worker 5 stopped. I0504 20:42:12.627987 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 17.391629ms to process 0 records I0504 20:42:12.630447 1 tasks_processing.go:74] worker 60 stopped. I0504 20:42:12.630604 1 recorder.go:75] Recording config/apiserver with fingerprint=f8414b5c88ed577f38dd23b6faeb8394c10753ab058747525d20e49592398970 I0504 20:42:12.630620 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 22.26676ms to process 1 records I0504 20:42:12.630729 1 tasks_processing.go:74] worker 11 stopped. E0504 20:42:12.630740 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0504 20:42:12.630746 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 22.050341ms to process 0 records I0504 20:42:12.631244 1 tasks_processing.go:74] worker 47 stopped. I0504 20:42:12.631329 1 recorder.go:75] Recording config/image with fingerprint=8ad830a774f99282ff3224c436a4f9add09ccce9975e1bdc25e28961ef42021b I0504 20:42:12.631344 1 gather.go:177] gatherer "clusterconfig" function "image" took 22.098425ms to process 1 records I0504 20:42:12.632584 1 tasks_processing.go:74] worker 1 stopped. I0504 20:42:12.632652 1 recorder.go:75] Recording config/proxy with fingerprint=312b4857dac379f6509373d646d75785664b37a7f992abbb43bb9752676750d1 I0504 20:42:12.632685 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 24.154252ms to process 1 records I0504 20:42:12.632915 1 tasks_processing.go:74] worker 8 stopped. I0504 20:42:12.632925 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 22.138445ms to process 0 records I0504 20:42:12.635217 1 tasks_processing.go:74] worker 18 stopped. I0504 20:42:12.635228 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 25.755848ms to process 0 records I0504 20:42:12.637365 1 tasks_processing.go:74] worker 14 stopped. I0504 20:42:12.638175 1 recorder.go:75] Recording config/node/ip-10-0-0-96.ec2.internal with fingerprint=90700579d172993207ad9136dcf10cde748b3c62169a75dcd4f1f59842bed5f2 I0504 20:42:12.638243 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 29.106008ms to process 1 records I0504 20:42:12.638564 1 tasks_processing.go:74] worker 12 stopped. I0504 20:42:12.638639 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0504 20:42:12.638681 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 30.824822ms to process 1 records I0504 20:42:12.638710 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 29.160294ms to process 0 records I0504 20:42:12.638737 1 tasks_processing.go:74] worker 52 stopped. I0504 20:42:12.639298 1 tasks_processing.go:74] worker 63 stopped. I0504 20:42:12.639311 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 31.183323ms to process 0 records I0504 20:42:12.639456 1 tasks_processing.go:74] worker 33 stopped. I0504 20:42:12.639471 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 30.021079ms to process 0 records I0504 20:42:12.639489 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 31.446046ms to process 0 records I0504 20:42:12.639513 1 tasks_processing.go:74] worker 59 stopped. I0504 20:42:12.639620 1 tasks_processing.go:74] worker 39 stopped. I0504 20:42:12.639633 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 31.057856ms to process 0 records I0504 20:42:12.639743 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0504 20:42:12.639753 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0504 20:42:12.639945 1 operator.go:288] started I0504 20:42:12.639967 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0504 20:42:12.640871 1 tasks_processing.go:74] worker 61 stopped. I0504 20:42:12.641065 1 recorder.go:75] Recording config/oauth with fingerprint=bdb8d4b6461d83950393f4ec40ab5bf77bd5c9ad3bf36f0d55d3b2a11307ce03 I0504 20:42:12.641078 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 31.5175ms to process 1 records I0504 20:42:12.641132 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=a3e6425636b78acff4bbb343de282073b270ee2ad459b9eb9ea8a4e90f0a3731 I0504 20:42:12.641145 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=c0731d23acf561e439332651bc11ebb83c66d5e5c21b749a16377037c8ef9741 I0504 20:42:12.641150 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 33.174714ms to process 2 records I0504 20:42:12.641156 1 tasks_processing.go:74] worker 41 stopped. I0504 20:42:12.647447 1 tasks_processing.go:74] worker 31 stopped. I0504 20:42:12.647460 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 38.315741ms to process 0 records I0504 20:42:12.650072 1 tasks_processing.go:74] worker 54 stopped. I0504 20:42:12.650088 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 40.711133ms to process 0 records I0504 20:42:12.650228 1 tasks_processing.go:74] worker 55 stopped. I0504 20:42:12.650323 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=d1dee04b315a13fc48757184040acb958ec675852d35c4114f5b101f1b5a5d24 I0504 20:42:12.650348 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=e0dfacad5e8ef534846cf5a5f947e2548818868b2b73d772fd939cdf1f3380ed I0504 20:42:12.650365 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=1045e87cf33f5800b8912fad85b33e61c6d6061f414bf3ddb4667008d53c250e I0504 20:42:12.650374 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 40.312807ms to process 3 records I0504 20:42:12.650417 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=701a234a59b497fe2931b6b8381600ab37067917662da5a7c85511798b7cd7db I0504 20:42:12.650424 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 41.097506ms to process 1 records I0504 20:42:12.650432 1 tasks_processing.go:74] worker 50 stopped. I0504 20:42:12.651686 1 tasks_processing.go:74] worker 15 stopped. I0504 20:42:12.651712 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 42.688915ms to process 0 records I0504 20:42:12.651915 1 tasks_processing.go:74] worker 35 stopped. I0504 20:42:12.651933 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 42.126669ms to process 0 records I0504 20:42:12.651972 1 tasks_processing.go:74] worker 53 stopped. I0504 20:42:12.652212 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=6bbd01e99bdca12fbf9a18fe19d8b6544c97e6a41d54047394ee01a1ec4bc16b I0504 20:42:12.652367 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=2b922b22b879d777cfbddff05cda0fcc39b96e6d1e16e6106988515ceb57fe7a I0504 20:42:12.652408 1 controller.go:212] Source scaController *sca.Controller is not ready I0504 20:42:12.652413 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=f611229711a1c46395a64a031ee997e76e3ba38dde05abcf45e13490fa1e76a2 I0504 20:42:12.652452 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=2a208bb7eccfd6aa3e5d74ac4e7778f608837869d3b39a3eb2421f99bf01f14d I0504 20:42:12.652477 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=4301df5e878acabe0bba319ecd9623edad371524f725bfa8bf4a935aeadddb6b I0504 20:42:12.652415 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0504 20:42:12.652489 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0504 20:42:12.652516 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0504 20:42:12.652520 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0504 20:42:12.652533 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=b873b98d6ea7239d73fccc18741dc5c825098118043542a791de3ab5538b14ec I0504 20:42:12.652536 1 controller.go:489] The operator is still being initialized I0504 20:42:12.652544 1 controller.go:512] The operator is healthy I0504 20:42:12.652560 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=ee367fc78b8ba14f25725904de92f4e447a023f4bb67d4f61f32bbe7caa876bb I0504 20:42:12.652596 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=28807e2ad5c9f60a671543d782e5617281715be5201f4bb10b1ed6fd3618f455 I0504 20:42:12.652620 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=64f721b21d9b5e34d67dc4f423d626ddf719a9bc59d32f03749c5440aad5223a I0504 20:42:12.652643 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=7a4c7030e855b37849501ee3936af7e08e8bf650fe765ff016cb3db856f602a5 I0504 20:42:12.652666 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=103478913785a61285d03b92c86c193518e8015af4770045eb078c753a7b83f2 I0504 20:42:12.652682 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 41.694574ms to process 11 records I0504 20:42:12.652769 1 tasks_processing.go:74] worker 57 stopped. I0504 20:42:12.652785 1 recorder.go:75] Recording config/featuregate with fingerprint=4c0042b2730fb8b309040bd99343f3e4842c381c20603ba82b5c4c4bef88a17d I0504 20:42:12.652791 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 44.169632ms to process 1 records I0504 20:42:12.652801 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 42.932817ms to process 0 records I0504 20:42:12.652810 1 tasks_processing.go:74] worker 51 stopped. I0504 20:42:12.652880 1 tasks_processing.go:74] worker 17 stopped. I0504 20:42:12.653410 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=659088f0061c0d780b702c794f3ede02c75dbfa3bab9e779e51399c4eb2f64a9 I0504 20:42:12.653623 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=adcd00389da022f3f6b1549d13c38bc845d0ac737d576c4849afae5be02fb995 I0504 20:42:12.653634 1 gather.go:177] gatherer "clusterconfig" function "crds" took 43.415494ms to process 2 records I0504 20:42:12.653715 1 tasks_processing.go:74] worker 56 stopped. I0504 20:42:12.653729 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=5090129fd975d5b184c159bf348f5b368e2cf3d6bd6ab81eea948ca68f7a0572 I0504 20:42:12.653754 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=9d209a93001dbc9a9155938a4a7d35f0db1133eb7281eaf4be336860e21fc37d I0504 20:42:12.653781 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=2495a296f317352aba347ccabedaf593d53e485eab2add65b78812acd1c85e38 I0504 20:42:12.653788 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 43.336853ms to process 3 records I0504 20:42:12.653869 1 tasks_processing.go:74] worker 21 stopped. I0504 20:42:12.653955 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=52778d30df9823f0953ed56060c5f607b9c402d84fc92548632659cc0c291a40 I0504 20:42:12.654129 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=88ca4b6317f6332d838547b0056816ee2bde4356f09729de0c63829d4d827e54 I0504 20:42:12.654141 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 43.842563ms to process 2 records W0504 20:42:12.657429 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 20:42:12.658172 1 tasks_processing.go:74] worker 16 stopped. I0504 20:42:12.658567 1 prometheus_rules.go:88] Prometheus rules successfully created I0504 20:42:12.659416 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-6kkgj with fingerprint=a9413cb23e73304a7fb4bb0d0bb9b2e564413b53dd54a85aeac8fabdbff7ad6a I0504 20:42:12.659465 1 recorder.go:75] Recording config/running_containers with fingerprint=305f427e5e5ba36b96a6b960afc036a12955be1f3c44c6ea05e93015c290e7fe I0504 20:42:12.659475 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 49.216936ms to process 2 records E0504 20:42:12.662616 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27d0be0502-2f99-41b1-ac54-fa19cf8c08a2%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.15:48215->172.30.0.10:53: read: connection refused I0504 20:42:12.662632 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27d0be0502-2f99-41b1-ac54-fa19cf8c08a2%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.15:48215->172.30.0.10:53: read: connection refused I0504 20:42:12.663428 1 tasks_processing.go:74] worker 2 stopped. I0504 20:42:12.663443 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 55.670495ms to process 0 records I0504 20:42:12.672957 1 tasks_processing.go:74] worker 6 stopped. E0504 20:42:12.672984 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0504 20:42:12.673003 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q3b3n94e4hdp4o9hv02mhjh2v21qm2j-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q3b3n94e4hdp4o9hv02mhjh2v21qm2j-primary-cert-bundle-secret" not found I0504 20:42:12.673409 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=d79189a076295ab517f76dab75739e783e8d1ca4f5c191cce4a2a53af67d0215 I0504 20:42:12.673427 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 63.573739ms to process 1 records I0504 20:42:12.676808 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0504 20:42:12.676828 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0504 20:42:12.676874 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0504 20:42:12.677329 1 tasks_processing.go:74] worker 9 stopped. E0504 20:42:12.677342 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0504 20:42:12.677348 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0504 20:42:12.677352 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0504 20:42:12.677361 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0504 20:42:12.677386 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0504 20:42:12.677393 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0504 20:42:12.677397 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0504 20:42:12.677400 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0504 20:42:12.677437 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0504 20:42:12.677444 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0504 20:42:12.677449 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 69.495433ms to process 7 records I0504 20:42:12.680595 1 base_controller.go:82] Caches are synced for ConfigController I0504 20:42:12.680609 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0504 20:42:12.685817 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:42:12.724411 1 tasks_processing.go:74] worker 44 stopped. I0504 20:42:12.724424 1 gather_logs.go:145] no pods in namespace were found I0504 20:42:12.724427 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 115.82862ms to process 0 records I0504 20:42:12.724436 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 114.28166ms to process 0 records I0504 20:42:12.724441 1 tasks_processing.go:74] worker 32 stopped. I0504 20:42:12.724550 1 tasks_processing.go:74] worker 38 stopped. I0504 20:42:12.724856 1 recorder.go:75] Recording config/version with fingerprint=86c4324e5bc8cd0914df3bfe784a9250cc1f356d259146de2c456abb892ea7b8 I0504 20:42:12.724870 1 recorder.go:75] Recording config/id with fingerprint=21dd60a73f25a1a5d7e31fc5d0cac91e6d2656dfdc4289edd0fd11701dc64cc9 I0504 20:42:12.724877 1 gather.go:177] gatherer "clusterconfig" function "version" took 116.716772ms to process 2 records I0504 20:42:12.726991 1 tasks_processing.go:74] worker 37 stopped. I0504 20:42:12.727024 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0504 20:42:12.727048 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 119.228101ms to process 1 records I0504 20:42:12.733488 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0504 20:42:12.737632 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.15:51182->172.30.0.10:53: read: connection refused I0504 20:42:12.737645 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.15:51182->172.30.0.10:53: read: connection refused I0504 20:42:12.740633 1 base_controller.go:82] Caches are synced for LoggingSyncer I0504 20:42:12.740645 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... W0504 20:42:13.656397 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 20:42:13.672733 1 tasks_processing.go:74] worker 58 stopped. I0504 20:42:13.672751 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 1.064951822s to process 0 records I0504 20:42:13.843082 1 gather_cluster_operator_pods_and_events.go:121] Found 12 pods with 14 containers I0504 20:42:13.843099 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1797558 bytes I0504 20:42:13.843130 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-xfxgd pod in namespace openshift-dns (previous: false). I0504 20:42:14.069161 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-xfxgd pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-xfxgd\" is waiting to start: ContainerCreating" I0504 20:42:14.069183 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-xfxgd\" is waiting to start: ContainerCreating" I0504 20:42:14.069192 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-xfxgd pod in namespace openshift-dns (previous: false). I0504 20:42:14.270830 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-xfxgd pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-xfxgd\" is waiting to start: ContainerCreating" I0504 20:42:14.270849 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-xfxgd\" is waiting to start: ContainerCreating" I0504 20:42:14.270860 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-hb54h pod in namespace openshift-dns (previous: false). I0504 20:42:14.275732 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0504 20:42:14.454719 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:42:14.454738 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-79f6d5f9f9-snqvt pod in namespace openshift-image-registry (previous: false). I0504 20:42:14.640785 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:42:14.640803 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7fdf5dd476-d47rm pod in namespace openshift-image-registry (previous: false). W0504 20:42:14.655511 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 20:42:14.847757 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-7fdf5dd476-d47rm pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-7fdf5dd476-d47rm\" is waiting to start: ContainerCreating" I0504 20:42:14.847778 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-7fdf5dd476-d47rm\" is waiting to start: ContainerCreating" I0504 20:42:14.847789 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7fdf5dd476-n2vb6 pod in namespace openshift-image-registry (previous: false). I0504 20:42:14.872613 1 tasks_processing.go:74] worker 43 stopped. I0504 20:42:14.872665 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=c937d5434f2ff4dc017f2ae5c856ca4f4a46d2528c568c6f8064f0fafdb70afb I0504 20:42:14.872695 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=979b9b5a2020046cafb60f065585483d28aecdef902569b3b8315539004d0ff6 I0504 20:42:14.872726 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0504 20:42:14.872751 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=3347405d97e139c9e933d317b350c8ed51a5fb84e9d9802fe5ddd960722d5752 I0504 20:42:14.872770 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0504 20:42:14.872792 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=65a022e3253e2f8ede3fe54ef2117713cd6c477ad274e6edd3b66a7fbae05660 I0504 20:42:14.872825 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=615b168e121806eb4e171694e1a6abf8a3f99f1348521f7fa94efd6878b48507 I0504 20:42:14.872863 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=d94840c5a2e1c7458074834cab370a0b0fe6f4b2caec1d256023ce7a7af9b3f5 I0504 20:42:14.872894 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=3badf25cec743635e699c204bf65dcb120519b7b4890c18790ada7a9ebeb2db3 I0504 20:42:14.872904 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=e5ff11d57817f84a678f6fa9565af55bd1120227c16a21933637ab62675a6d70 I0504 20:42:14.872920 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=b58795ed0a2e86976c38c5a5d50741cf2727c635b8831b8ad0b363d5b4579226 I0504 20:42:14.872930 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0504 20:42:14.872946 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=81bbc97037c2eccb373d170b6df65121c8d294a53d5ca3f957f0731784a982f3 I0504 20:42:14.872954 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0504 20:42:14.872970 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=3bef810f646378da54f152438578645ef81f22c453c837fb108cd0c68b9f10f8 I0504 20:42:14.872981 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0504 20:42:14.873000 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=ac467e2b91dd657ce131c914e199482492ee2d881e9746243b8548a3bd8e2cb8 I0504 20:42:14.873010 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0504 20:42:14.873029 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=c2744ac899eef501ddddb1f973fe93983287dbef688f14b0369d5acb1dd08736 I0504 20:42:14.873149 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=7cb04ca2430c2f23ade49d72ed53b84c18f4ef26fbb5360bf614bde51ec0efd9 I0504 20:42:14.873159 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0504 20:42:14.873166 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0504 20:42:14.873187 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0504 20:42:14.873207 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=71474f799d27b10059b1d518a9854f458ed3c95831d9b8b70038d0ede1260371 I0504 20:42:14.873232 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=3a32ab2c95c676054c38e5cb1ac18f5e4529bda4dcb7e4114c3db4714b69d3a3 I0504 20:42:14.873242 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0504 20:42:14.873257 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=cebf50b576d31702dd09fc3bae1ed920b259e9ab8b62e78b0e68ff115016d862 I0504 20:42:14.873267 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0504 20:42:14.873280 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=d25b1229da25d75f689e00fb9c51ee55b2357d6335839d60ff97f513ce28c8a6 I0504 20:42:14.873295 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=9c9c9ef31f0affe08e7a0f967142cb180646d88540ad45f3f2b24b3746fb7580 I0504 20:42:14.873309 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=f8a8a95d6eeb96eb506f6c24ab928a87cc494f725377e5f94c284582cf02c964 I0504 20:42:14.873324 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=aa91969532c7becb51b2cf8b845ce664e6e4e63161e09c2045875511ebb6e52e I0504 20:42:14.873358 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=3554ea67ed8443af60ad5b9810b57bb303b33e5b68a0988cf268ad3379c61521 I0504 20:42:14.873369 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0504 20:42:14.873392 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=b4f8aa3c20b448eda6ae275e4dedd1d73c26e9a9bcfbc7cae53eae6913d6de89 I0504 20:42:14.873408 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0504 20:42:14.873418 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0504 20:42:14.873424 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.263070834s to process 37 records I0504 20:42:15.041544 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:42:15.041563 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-rwwxd pod in namespace openshift-image-registry (previous: false). I0504 20:42:15.246708 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:42:15.246728 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6d5b4dd9-pzcgj pod in namespace openshift-ingress (previous: false). I0504 20:42:15.446191 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-6d5b4dd9-pzcgj pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-6d5b4dd9-pzcgj\" is waiting to start: ContainerCreating" I0504 20:42:15.446210 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-6d5b4dd9-pzcgj\" is waiting to start: ContainerCreating" I0504 20:42:15.446222 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-79fd9bbbd4-82zrd pod in namespace openshift-ingress (previous: false). I0504 20:42:15.646422 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-79fd9bbbd4-82zrd pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-79fd9bbbd4-82zrd\" is waiting to start: ContainerCreating" I0504 20:42:15.646441 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-79fd9bbbd4-82zrd\" is waiting to start: ContainerCreating" I0504 20:42:15.646451 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-79fd9bbbd4-sflqq pod in namespace openshift-ingress (previous: false). W0504 20:42:15.656172 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 20:42:15.841222 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:42:15.841242 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-c7lh2 pod in namespace openshift-ingress-canary (previous: false). I0504 20:42:16.047587 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-c7lh2 pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-c7lh2\" is waiting to start: ContainerCreating" I0504 20:42:16.047605 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-c7lh2\" is waiting to start: ContainerCreating" I0504 20:42:16.047617 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for migrator container migrator-7d5f866c57-ftw2q pod in namespace openshift-kube-storage-version-migrator (previous: false). I0504 20:42:16.246927 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for graceful-termination container migrator-7d5f866c57-ftw2q pod in namespace openshift-kube-storage-version-migrator (previous: false). I0504 20:42:16.446227 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-storage-version-migrator-operator container kube-storage-version-migrator-operator-74848b4cb9-xxt8c pod in namespace openshift-kube-storage-version-migrator-operator (previous: false). I0504 20:42:16.649429 1 tasks_processing.go:74] worker 46 stopped. I0504 20:42:16.649526 1 recorder.go:75] Recording events/openshift-dns with fingerprint=7f961325ead81cda8155f15fcde0771c7e55ca24b6054c358b8babb238a0ea94 I0504 20:42:16.649597 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=812e2ac3ed90f8f2011ca871ee0a566bb6f82e01151d3761067040ca6223717d I0504 20:42:16.649625 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=31cfa56693169ec0842d7bbd6fc4a6ad20b7281ef6e9cef0677c6ff694ee9d13 I0504 20:42:16.649668 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=3f2ecc45e4a532dd7496877a9e8a5282e6a74be48499916027c4e938b43f828f I0504 20:42:16.649699 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=abc02084959143d6cc51d403588b6fb396a8182f495214dc11072045bb4ea373 I0504 20:42:16.649724 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator with fingerprint=2b4ba9094cc3ae4338e5dc75aa855451a2307081bf43c27b6e0d86481b4ddc50 I0504 20:42:16.649778 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator-operator with fingerprint=952ca8e87db6844a7fb25ed4d76d1138c576b2114bc8f6f8e0097de353132f86 I0504 20:42:16.649794 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-ftw2q/migrator_current.log with fingerprint=d34d85188196979d9b6d5347c480045543db0398099d441632610a50e8a1e14b I0504 20:42:16.649801 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-ftw2q/graceful-termination_current.log with fingerprint=6a715262ae93ac3dcb13fb984d833ee8c83b74845225f784a96872c33f5f76a4 I0504 20:42:16.649875 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator-operator/logs/kube-storage-version-migrator-operator-74848b4cb9-xxt8c/kube-storage-version-migrator-operator_current.log with fingerprint=82e03c170ee483b1677bf6dee3eb97f81364ef575de52ba367c99dce4d2cb773 I0504 20:42:16.649884 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 4.041676957s to process 10 records W0504 20:42:16.655882 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0504 20:42:17.652002 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0504 20:42:17.652041 1 tasks_processing.go:74] worker 34 stopped. E0504 20:42:17.652052 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0504 20:42:17.652064 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0504 20:42:17.652081 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0504 20:42:17.652092 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.042814385s to process 1 records I0504 20:42:22.189245 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:42:22.572242 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:42:22.776097 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:42:25.532335 1 tasks_processing.go:74] worker 48 stopped. I0504 20:42:25.532378 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0504 20:42:25.532392 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.922859786s to process 1 records I0504 20:42:26.215711 1 tasks_processing.go:74] worker 29 stopped. I0504 20:42:26.215989 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=f88634c9e6ef34943071c11cac352a5958ad8eb9df87284bbed36f89ce5bfbb1 I0504 20:42:26.216011 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.607184987s to process 1 records E0504 20:42:26.216066 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.608s with: function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error" I0504 20:42:26.217172 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error I0504 20:42:26.217187 1 periodic.go:209] Running workloads gatherer I0504 20:42:26.217202 1 tasks_processing.go:45] number of workers: 2 I0504 20:42:26.217211 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 20:42:26.217215 1 tasks_processing.go:71] worker 1 working on workload_info task. I0504 20:42:26.217222 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 20:42:26.217234 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0504 20:42:26.235876 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 19s for image data I0504 20:42:26.243770 1 tasks_processing.go:74] worker 0 stopped. I0504 20:42:26.243785 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 26.521955ms to process 0 records I0504 20:42:26.245030 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (10ms) I0504 20:42:26.253278 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (8ms) I0504 20:42:26.265127 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (12ms) I0504 20:42:26.274824 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (10ms) I0504 20:42:26.283423 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (9ms) I0504 20:42:26.291938 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (9ms) I0504 20:42:26.300053 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (8ms) I0504 20:42:26.309165 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (9ms) I0504 20:42:26.317392 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (8ms) I0504 20:42:26.333165 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (16ms) I0504 20:42:26.344235 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (11ms) I0504 20:42:26.445410 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (101ms) I0504 20:42:26.544512 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (99ms) I0504 20:42:26.645122 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (101ms) I0504 20:42:26.745334 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (100ms) I0504 20:42:26.845617 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0504 20:42:26.944595 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (99ms) I0504 20:42:27.045952 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (101ms) I0504 20:42:27.145674 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (100ms) I0504 20:42:27.245330 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (100ms) I0504 20:42:27.346125 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (101ms) I0504 20:42:27.445381 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (99ms) I0504 20:42:27.545255 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (100ms) I0504 20:42:27.646274 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (101ms) I0504 20:42:27.745522 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (99ms) I0504 20:42:27.845070 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (100ms) I0504 20:42:27.945381 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0504 20:42:28.046485 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (101ms) I0504 20:42:28.145326 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (99ms) I0504 20:42:28.245277 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0504 20:42:28.353933 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (109ms) I0504 20:42:28.353960 1 tasks_processing.go:74] worker 1 stopped. E0504 20:42:28.353971 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0504 20:42:28.354264 1 recorder.go:75] Recording config/workload_info with fingerprint=aeb60eae57962a10f2f09e172e5e5ce87f79a1885d659bb162e2d5709730b0ab I0504 20:42:28.354279 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.136738592s to process 1 records E0504 20:42:28.354301 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.137s with: function \"workload_info\" failed with an error" I0504 20:42:28.355405 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0504 20:42:28.355419 1 periodic.go:209] Running conditional gatherer I0504 20:42:28.362849 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0504 20:42:28.370903 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.15:44003->172.30.0.10:53: read: connection refused E0504 20:42:28.371135 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:42:28.371190 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0504 20:42:28.377228 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0504 20:42:28.377242 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:42:28.377248 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:42:28.377251 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:42:28.377255 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:42:28.377258 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:42:28.377261 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:42:28.377264 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:42:28.377267 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:42:28.377269 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0504 20:42:28.377284 1 tasks_processing.go:45] number of workers: 3 I0504 20:42:28.377294 1 tasks_processing.go:69] worker 2 listening for tasks. I0504 20:42:28.377298 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0504 20:42:28.377304 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 20:42:28.377316 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0504 20:42:28.377318 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 20:42:28.377322 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0504 20:42:28.377326 1 tasks_processing.go:74] worker 1 stopped. I0504 20:42:28.377380 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0504 20:42:28.377392 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 789ns to process 1 records I0504 20:42:28.377423 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0504 20:42:28.377432 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.56µs to process 1 records I0504 20:42:28.377437 1 tasks_processing.go:74] worker 0 stopped. I0504 20:42:28.377565 1 tasks_processing.go:74] worker 2 stopped. I0504 20:42:28.377577 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 235.678µs to process 0 records I0504 20:42:28.377595 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.15:44003->172.30.0.10:53: read: connection refused I0504 20:42:28.377612 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0504 20:42:28.402976 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=cd720d9f629f5e7e9043e347edac8d4a5fcf91cfb7eae2034fd98c40baefcb86 I0504 20:42:28.403109 1 diskrecorder.go:70] Writing 104 records to /var/lib/insights-operator/insights-2026-05-04-204228.tar.gz I0504 20:42:28.409625 1 diskrecorder.go:51] Wrote 104 records to disk in 6ms I0504 20:42:28.409652 1 periodic.go:278] Gathering cluster info every 2h0m0s I0504 20:42:28.409665 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0504 20:42:33.350635 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:43:36.659539 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="8b9fea271e41bb21440f1315b31d0e4203d11f3b8d12ab1dfdf11a68a16ec4cc") W0504 20:43:36.659573 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0504 20:43:36.659619 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="397291f3668138f64ead678da5ec7ce635096ef006e51e5eedfa63865b6e02aa") I0504 20:43:36.659623 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0504 20:43:36.659651 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0504 20:43:36.659683 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0504 20:43:36.659699 1 base_controller.go:181] Shutting down LoggingSyncer ... I0504 20:43:36.659709 1 genericapiserver.go:651] "[graceful-termination] not going to wait for active watch request(s) to drain" I0504 20:43:36.659716 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="3ac72fcbf9f66930ae825901e1d3f5dc04941fe344d76d963dd3d90b1aaa105b") I0504 20:43:36.659725 1 periodic.go:170] Shutting down I0504 20:43:36.659720 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0504 20:43:36.659736 1 requestheader_controller.go:194] Shutting down RequestHeaderAuthRequestController I0504 20:43:36.659736 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0504 20:43:36.659747 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0504 20:43:36.659748 1 base_controller.go:181] Shutting down ConfigController ... I0504 20:43:36.659758 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0504 20:43:36.659761 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0504 20:43:36.659775 1 base_controller.go:113] All ConfigController workers have been terminated I0504 20:43:36.659769 1 dynamic_serving_content.go:149] "Shutting down controller" name="serving-cert::/tmp/serving-cert-2990929576/tls.crt::/tmp/serving-cert-2990929576/tls.key" I0504 20:43:36.659789 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0504 20:43:36.659809 1 tlsconfig.go:258] "Shutting down DynamicServingCertificateController"