W0704 18:15:20.952060 1 cmd.go:257] Using insecure, self-signed certificates I0704 18:15:21.452944 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 18:15:21.453309 1 observer_polling.go:159] Starting file observer I0704 18:15:22.233653 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0704 18:15:22.233876 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0704 18:15:22.234478 1 secure_serving.go:57] Forcing use of http/1.1 only W0704 18:15:22.234500 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0704 18:15:22.234505 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0704 18:15:22.234510 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0704 18:15:22.234513 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0704 18:15:22.234515 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0704 18:15:22.234518 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0704 18:15:22.234576 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0704 18:15:22.238631 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0704 18:15:22.238633 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"eb011c49-7ac5-47c2-b7a3-0258d2158b70", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0704 18:15:22.239601 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0704 18:15:22.239684 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0704 18:15:22.239615 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0704 18:15:22.239640 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0704 18:15:22.239715 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0704 18:15:22.239721 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0704 18:15:22.240022 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-925379388/tls.crt::/tmp/serving-cert-925379388/tls.key" I0704 18:15:22.240349 1 secure_serving.go:213] Serving securely on [::]:8443 I0704 18:15:22.240381 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0704 18:15:22.244125 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0704 18:15:22.244147 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0704 18:15:22.244262 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0704 18:15:22.250565 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0704 18:15:22.250591 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0704 18:15:22.256509 1 secretconfigobserver.go:119] support secret does not exist I0704 18:15:22.261768 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0704 18:15:22.266755 1 secretconfigobserver.go:119] support secret does not exist I0704 18:15:22.270263 1 recorder.go:161] Pruning old reports every 5h57m37s, max age is 288h0m0s I0704 18:15:22.276595 1 periodic.go:209] Running clusterconfig gatherer I0704 18:15:22.276659 1 tasks_processing.go:45] number of workers: 32 I0704 18:15:22.276666 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0704 18:15:22.276595 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0704 18:15:22.276681 1 insightsreport.go:296] Starting report retriever I0704 18:15:22.276688 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0704 18:15:22.276696 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0704 18:15:22.276704 1 tasks_processing.go:69] worker 31 listening for tasks. I0704 18:15:22.276711 1 tasks_processing.go:71] worker 31 working on active_alerts task. I0704 18:15:22.276710 1 tasks_processing.go:69] worker 5 listening for tasks. I0704 18:15:22.276711 1 tasks_processing.go:69] worker 0 listening for tasks. I0704 18:15:22.276719 1 tasks_processing.go:69] worker 1 listening for tasks. I0704 18:15:22.276714 1 tasks_processing.go:69] worker 18 listening for tasks. I0704 18:15:22.276724 1 tasks_processing.go:69] worker 2 listening for tasks. I0704 18:15:22.276730 1 tasks_processing.go:69] worker 4 listening for tasks. I0704 18:15:22.276724 1 tasks_processing.go:69] worker 3 listening for tasks. I0704 18:15:22.276722 1 tasks_processing.go:69] worker 6 listening for tasks. I0704 18:15:22.276728 1 tasks_processing.go:69] worker 24 listening for tasks. I0704 18:15:22.276741 1 tasks_processing.go:69] worker 29 listening for tasks. I0704 18:15:22.276742 1 tasks_processing.go:69] worker 20 listening for tasks. I0704 18:15:22.276739 1 tasks_processing.go:69] worker 27 listening for tasks. I0704 18:15:22.276750 1 tasks_processing.go:69] worker 22 listening for tasks. I0704 18:15:22.276750 1 tasks_processing.go:69] worker 28 listening for tasks. I0704 18:15:22.276753 1 tasks_processing.go:69] worker 26 listening for tasks. I0704 18:15:22.276740 1 tasks_processing.go:69] worker 8 listening for tasks. I0704 18:15:22.276757 1 tasks_processing.go:69] worker 23 listening for tasks. I0704 18:15:22.276758 1 tasks_processing.go:69] worker 21 listening for tasks. I0704 18:15:22.276765 1 tasks_processing.go:69] worker 15 listening for tasks. I0704 18:15:22.276768 1 tasks_processing.go:69] worker 13 listening for tasks. I0704 18:15:22.276764 1 tasks_processing.go:69] worker 14 listening for tasks. I0704 18:15:22.276773 1 tasks_processing.go:71] worker 15 working on container_runtime_configs task. I0704 18:15:22.276777 1 tasks_processing.go:71] worker 22 working on support_secret task. I0704 18:15:22.276779 1 tasks_processing.go:71] worker 3 working on pdbs task. I0704 18:15:22.276782 1 tasks_processing.go:69] worker 11 listening for tasks. I0704 18:15:22.276781 1 tasks_processing.go:69] worker 16 listening for tasks. I0704 18:15:22.276735 1 tasks_processing.go:69] worker 19 listening for tasks. I0704 18:15:22.276746 1 tasks_processing.go:69] worker 25 listening for tasks. I0704 18:15:22.276747 1 tasks_processing.go:69] worker 9 listening for tasks. I0704 18:15:22.276793 1 tasks_processing.go:71] worker 28 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0704 18:15:22.276798 1 tasks_processing.go:71] worker 8 working on jaegers task. I0704 18:15:22.276799 1 tasks_processing.go:71] worker 2 working on openstack_dataplanenodesets task. I0704 18:15:22.276778 1 tasks_processing.go:69] worker 30 listening for tasks. I0704 18:15:22.276887 1 tasks_processing.go:71] worker 23 working on sap_datahubs task. I0704 18:15:22.276894 1 tasks_processing.go:71] worker 24 working on openstack_version task. I0704 18:15:22.277037 1 tasks_processing.go:71] worker 6 working on openshift_logging task. I0704 18:15:22.276772 1 tasks_processing.go:71] worker 21 working on silenced_alerts task. I0704 18:15:22.277048 1 tasks_processing.go:71] worker 29 working on machines task. W0704 18:15:22.277258 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 18:15:22.277292 1 tasks_processing.go:71] worker 21 working on mutating_webhook_configurations task. I0704 18:15:22.277316 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 80.025µs to process 0 records I0704 18:15:22.276766 1 tasks_processing.go:69] worker 12 listening for tasks. I0704 18:15:22.276771 1 tasks_processing.go:71] worker 4 working on ingress task. I0704 18:15:22.277447 1 tasks_processing.go:71] worker 12 working on nodenetworkstates task. I0704 18:15:22.276732 1 tasks_processing.go:69] worker 7 listening for tasks. I0704 18:15:22.277534 1 tasks_processing.go:71] worker 7 working on image task. I0704 18:15:22.276774 1 tasks_processing.go:71] worker 27 working on lokistack task. I0704 18:15:22.276783 1 tasks_processing.go:71] worker 5 working on schedulers task. I0704 18:15:22.276789 1 tasks_processing.go:71] worker 26 working on version task. I0704 18:15:22.276788 1 tasks_processing.go:71] worker 0 working on proxies task. I0704 18:15:22.276793 1 tasks_processing.go:71] worker 1 working on aggregated_monitoring_cr_names task. I0704 18:15:22.276796 1 tasks_processing.go:71] worker 18 working on nodes task. I0704 18:15:22.276756 1 tasks_processing.go:69] worker 10 listening for tasks. I0704 18:15:22.278019 1 tasks_processing.go:71] worker 10 working on certificate_signing_requests task. I0704 18:15:22.277060 1 tasks_processing.go:71] worker 30 working on machine_configs task. I0704 18:15:22.277055 1 tasks_processing.go:71] worker 20 working on networks task. W0704 18:15:22.276755 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 18:15:22.278260 1 tasks_processing.go:71] worker 31 working on cluster_apiserver task. I0704 18:15:22.277067 1 tasks_processing.go:71] worker 16 working on tsdb_status task. I0704 18:15:22.277069 1 tasks_processing.go:69] worker 17 listening for tasks. I0704 18:15:22.277072 1 tasks_processing.go:71] worker 13 working on sap_pods task. I0704 18:15:22.277074 1 tasks_processing.go:71] worker 19 working on machine_autoscalers task. I0704 18:15:22.277079 1 tasks_processing.go:71] worker 25 working on cost_management_metrics_configs task. I0704 18:15:22.277076 1 tasks_processing.go:71] worker 11 working on qemu_kubevirt_launcher_logs task. I0704 18:15:22.277076 1 tasks_processing.go:71] worker 14 working on openshift_machine_api_events task. I0704 18:15:22.277083 1 tasks_processing.go:71] worker 9 working on clusterroles task. I0704 18:15:22.278296 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.52305ms to process 0 records I0704 18:15:22.278324 1 tasks_processing.go:71] worker 17 working on nodenetworkconfigurationpolicies task. W0704 18:15:22.278346 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 18:15:22.278356 1 tasks_processing.go:71] worker 16 working on service_accounts task. I0704 18:15:22.278913 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 24.538µs to process 0 records I0704 18:15:22.282730 1 tasks_processing.go:71] worker 15 working on oauths task. I0704 18:15:22.282774 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 5.948462ms to process 0 records I0704 18:15:22.282786 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 5.871086ms to process 0 records I0704 18:15:22.282794 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 5.979458ms to process 0 records I0704 18:15:22.282802 1 tasks_processing.go:71] worker 2 working on validating_webhook_configurations task. I0704 18:15:22.282858 1 tasks_processing.go:71] worker 23 working on operators_pods_and_events task. I0704 18:15:22.283179 1 tasks_processing.go:71] worker 8 working on image_registries task. I0704 18:15:22.283190 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 6.373682ms to process 0 records I0704 18:15:22.283422 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0704 18:15:22.283450 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0704 18:15:22.283455 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0704 18:15:22.283458 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0704 18:15:22.283473 1 controller.go:489] The operator is still being initialized I0704 18:15:22.283479 1 controller.go:512] The operator is healthy I0704 18:15:22.287997 1 tasks_processing.go:71] worker 24 working on monitoring_persistent_volumes task. I0704 18:15:22.288013 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 11.091392ms to process 0 records I0704 18:15:22.288029 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 10.899145ms to process 0 records I0704 18:15:22.288057 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 10.42701ms to process 0 records I0704 18:15:22.288137 1 tasks_processing.go:71] worker 27 working on openstack_dataplanedeployments task. I0704 18:15:22.288166 1 tasks_processing.go:71] worker 6 working on image_pruners task. I0704 18:15:22.288166 1 tasks_processing.go:71] worker 12 working on crds task. I0704 18:15:22.288186 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 10.695042ms to process 0 records I0704 18:15:22.288398 1 tasks_processing.go:71] worker 3 working on sap_config task. I0704 18:15:22.288526 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=6c29ec36d93bff9026112ff8c9e138eef45dc7f4965e2f5ed814993cfd326dd6 I0704 18:15:22.288569 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=673f0e5a839c0f2cb6630455bac92c7755e0f9ea1b6723222071dc07485924d2 I0704 18:15:22.288599 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=c8914b71366735e331c84f9cf3f0a987073aa4380de7d3d229ff4d4e788043a5 I0704 18:15:22.288613 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 11.459809ms to process 3 records E0704 18:15:22.288629 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0704 18:15:22.288638 1 gather.go:177] gatherer "clusterconfig" function "machines" took 11.063654ms to process 0 records I0704 18:15:22.288648 1 tasks_processing.go:71] worker 29 working on olm_operators task. I0704 18:15:22.297591 1 tasks_processing.go:71] worker 17 working on dvo_metrics task. I0704 18:15:22.297620 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 19.242548ms to process 0 records I0704 18:15:22.297669 1 gather_logs.go:145] no pods in namespace were found I0704 18:15:22.297783 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 19.227967ms to process 0 records I0704 18:15:22.297793 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 19.086117ms to process 0 records I0704 18:15:22.297797 1 tasks_processing.go:71] worker 25 working on storage_classes task. I0704 18:15:22.297807 1 tasks_processing.go:71] worker 19 working on overlapping_namespace_uids task. I0704 18:15:22.297799 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 19.192945ms to process 0 records I0704 18:15:22.297874 1 tasks_processing.go:71] worker 13 working on node_logs task. I0704 18:15:22.298096 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 19.166421ms to process 0 records I0704 18:15:22.298173 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 19.258334ms to process 0 records I0704 18:15:22.298229 1 tasks_processing.go:71] worker 11 working on metrics task. W0704 18:15:22.298285 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 18:15:22.298316 1 recorder.go:75] Recording config/proxy with fingerprint=b46d11aa88c17b8877a29183bbc76c4f636f3d5fd1a62dd2fc315252dba666ea I0704 18:15:22.298365 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 20.067458ms to process 1 records I0704 18:15:22.298321 1 tasks_processing.go:71] worker 0 working on machine_healthchecks task. I0704 18:15:22.298474 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=84025be9854e5b214bf260cbab0f8a57df5baef71816a31a04152defe2aa7040 I0704 18:15:22.298511 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 20.400854ms to process 1 records I0704 18:15:22.298539 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 54.939µs to process 0 records I0704 18:15:22.298575 1 tasks_processing.go:71] worker 11 working on feature_gates task. I0704 18:15:22.298900 1 tasks_processing.go:71] worker 14 working on config_maps task. I0704 18:15:22.299016 1 tasks_processing.go:71] worker 5 working on install_plans task. I0704 18:15:22.299802 1 tasks_processing.go:71] worker 4 working on operators task. I0704 18:15:22.300393 1 recorder.go:75] Recording config/ingress with fingerprint=0fd0306713b8f6c8af8a87f6abdad235d7dfa6b5ce1d8089e36dcda0c5149aeb I0704 18:15:22.300462 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 22.40656ms to process 1 records I0704 18:15:22.308319 1 tasks_processing.go:71] worker 27 working on pod_network_connectivity_checks task. I0704 18:15:22.308370 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 20.160549ms to process 0 records I0704 18:15:22.308554 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 19.93558ms to process 0 records I0704 18:15:22.308566 1 tasks_processing.go:71] worker 3 working on machine_config_pools task. I0704 18:15:22.308776 1 tasks_processing.go:71] worker 15 working on storage_cluster task. I0704 18:15:22.309097 1 recorder.go:75] Recording config/oauth with fingerprint=a64ba60ba36408887c25258725b23f3375e0f690737bcc8e171c9d451c365cc4 I0704 18:15:22.309120 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 26.025978ms to process 1 records I0704 18:15:22.309227 1 tasks_processing.go:71] worker 18 working on ceph_cluster task. I0704 18:15:22.309555 1 recorder.go:75] Recording config/node/ip-10-0-0-17.ec2.internal with fingerprint=47c79fb7baa2a17e48b8379538f9bc2c4202bcf1a24c15ddd1c6e62b22beb7b3 I0704 18:15:22.309629 1 recorder.go:75] Recording config/node/ip-10-0-1-36.ec2.internal with fingerprint=c643f3fe6243f3c9985396c5e3570ee774bc8bbecf9b75c91c98fb0be3c95eba I0704 18:15:22.309697 1 recorder.go:75] Recording config/node/ip-10-0-2-229.ec2.internal with fingerprint=8a495b8e44e35c2b33211ed4f8e59469c6785ed849add622d77ef239d709bc82 I0704 18:15:22.309711 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 31.334396ms to process 3 records I0704 18:15:22.310348 1 tasks_processing.go:71] worker 22 working on infrastructures task. E0704 18:15:22.310363 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0704 18:15:22.310376 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 33.55654ms to process 0 records I0704 18:15:22.310949 1 tasks_processing.go:71] worker 29 working on machine_sets task. I0704 18:15:22.310997 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 22.292312ms to process 0 records I0704 18:15:22.311555 1 tasks_processing.go:71] worker 31 working on authentication task. I0704 18:15:22.311633 1 recorder.go:75] Recording config/apiserver with fingerprint=bb7caaeba836f021e7694105f2defbd031519b02835fb96fd2506ea7a79195c4 I0704 18:15:22.311681 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 32.682875ms to process 1 records I0704 18:15:22.311842 1 tasks_processing.go:71] worker 20 working on ingress_certificates task. I0704 18:15:22.311930 1 recorder.go:75] Recording config/network with fingerprint=dddaf88cedfe0f972e8a0aa132dab2cea82647d0ad4f982eeeb51edf7c500ba4 I0704 18:15:22.311972 1 gather.go:177] gatherer "clusterconfig" function "networks" took 32.993032ms to process 1 records I0704 18:15:22.312230 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=c8dd1565a22562a4e5adab28befaf5fbd935fd558455a9643dea275c421b22bd I0704 18:15:22.312306 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=ad771eb4e6fe58b4c3fcc32e086be8eaf72705f7a8b2d76d29192bc2f0fd09ac I0704 18:15:22.313017 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=19d0835800f6a22f656f3b2fb8cd2eec12f0b635c27495d686ee0d414c9e8016 I0704 18:15:22.313232 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 33.774726ms to process 3 records I0704 18:15:22.313263 1 tasks_processing.go:71] worker 21 working on container_images task. I0704 18:15:22.313304 1 tasks_processing.go:71] worker 7 working on openstack_controlplanes task. I0704 18:15:22.313492 1 recorder.go:75] Recording config/image with fingerprint=54765c425df727ff6bd713d6443d1a01e42d27c501f869d803c443965819cf01 I0704 18:15:22.313512 1 gather.go:177] gatherer "clusterconfig" function "image" took 33.751114ms to process 1 records I0704 18:15:22.313724 1 tasks_processing.go:74] worker 2 stopped. I0704 18:15:22.313971 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=73015631e2809537003338d80b4c98dcb14bf38baf33231e6edff804e2d3b67c I0704 18:15:22.314274 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=4668c20c29054c97dc69e6d3b2ab77c9f3a9f6f9fe640fb9ba8afada6dc71ccf I0704 18:15:22.314332 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=b7f206b5fb9eb1ad717778f54aa3932e018d1197604379af79f45e1ffc4d0d43 I0704 18:15:22.314411 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=0c4c5fadcae02583bebf2ccd252706fb915522abc686a5928c63d8f05379083e I0704 18:15:22.314481 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=230f268f89c183258b25d9d3fc42710a639562c51dbfb64d12a63b5d14379f76 I0704 18:15:22.314550 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=a5a3fcf32d32b12d6cce4dbdfa8c4bd45420c8beb54bbb57d38b83b84234e0ce I0704 18:15:22.314663 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=1283f9874d5f5844316d1cc2aded0402c4b2b8727c839fc0a1c937f1e3b5fe42 I0704 18:15:22.314754 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=fd6eb488787532c53494a56633cc22dacb225a0ca908ec835db988943d4df1d5 I0704 18:15:22.314870 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=cd48e4ec08330c875e86ffacd576f5cd7c4bb6f91c5971ce199ba805440d0471 I0704 18:15:22.314967 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=b6e58de7fe5b86cc53c1d6a13308badbc32b93eb80a7e8ad81745f6110e75817 I0704 18:15:22.315061 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=9731806e82a1fc9410fc59553be361ff9f5685d04de8fb6f2d1831cf53c46e89 I0704 18:15:22.315093 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 28.585171ms to process 11 records I0704 18:15:22.315638 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=18593766e2afdfb9ccc82085fb60522a0319cdd75712cd9e0920855eee972a9a I0704 18:15:22.315681 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 23.391683ms to process 1 records I0704 18:15:22.315882 1 tasks_processing.go:74] worker 6 stopped. I0704 18:15:22.315935 1 tasks_processing.go:74] worker 8 stopped. I0704 18:15:22.316990 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=fd21ca41917c7c769386c3688a0de591f784fae3250d594b6323e1f960618a47 I0704 18:15:22.317072 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 28.560637ms to process 1 records I0704 18:15:22.317087 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 24.685472ms to process 0 records I0704 18:15:22.317094 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 36.981035ms to process 0 records E0704 18:15:22.317102 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0704 18:15:22.317113 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 8.383034ms to process 0 records E0704 18:15:22.317120 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0704 18:15:22.317127 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 18.425164ms to process 0 records I0704 18:15:22.317137 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 8.068755ms to process 0 records I0704 18:15:22.317145 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 7.647878ms to process 0 records I0704 18:15:22.317153 1 tasks_processing.go:74] worker 18 stopped. I0704 18:15:22.317160 1 tasks_processing.go:74] worker 24 stopped. I0704 18:15:22.317165 1 tasks_processing.go:74] worker 10 stopped. I0704 18:15:22.317170 1 tasks_processing.go:74] worker 27 stopped. I0704 18:15:22.317175 1 tasks_processing.go:74] worker 0 stopped. I0704 18:15:22.317179 1 tasks_processing.go:74] worker 15 stopped. I0704 18:15:22.317205 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0704 18:15:22.317294 1 operator.go:288] started I0704 18:15:22.317317 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0704 18:15:22.317332 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0704 18:15:22.333637 1 tasks_processing.go:74] worker 29 stopped. I0704 18:15:22.333666 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 22.468946ms to process 0 records I0704 18:15:22.333936 1 tasks_processing.go:74] worker 7 stopped. I0704 18:15:22.333957 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 20.576342ms to process 0 records I0704 18:15:22.334222 1 tasks_processing.go:74] worker 11 stopped. I0704 18:15:22.334397 1 recorder.go:75] Recording config/featuregate with fingerprint=f961157cd643252a31e5cc470168e0a78f3352575a7502f6ba76e660b1682373 I0704 18:15:22.334454 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 35.619165ms to process 1 records I0704 18:15:22.334582 1 tasks_processing.go:74] worker 31 stopped. I0704 18:15:22.334784 1 recorder.go:75] Recording config/authentication with fingerprint=8c89ef6df53352ad2a03efe50c63d81c100184fd47cbe7e441269a9fcf11c9e4 I0704 18:15:22.334860 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 22.81811ms to process 1 records I0704 18:15:22.334984 1 tasks_processing.go:74] worker 9 stopped. I0704 18:15:22.335098 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0704 18:15:22.335133 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=dc48584c24606e068c1bbd2ca259b85ee2c831dfd129ad9fd28af0795aa37e67 I0704 18:15:22.335676 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=6faa21a1c65caf66158cb5178ee0ef107da85ad0a8f1353cdfe184a3268324f5 I0704 18:15:22.335722 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 56.192687ms to process 2 records I0704 18:15:22.335138 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0704 18:15:22.335978 1 controller.go:212] Source scaController *sca.Controller is not ready I0704 18:15:22.336005 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0704 18:15:22.336013 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0704 18:15:22.336225 1 tasks_processing.go:74] worker 22 stopped. I0704 18:15:22.337250 1 recorder.go:75] Recording config/infrastructure with fingerprint=4a619cd140253c756263742ebf7ef069f68ad73e211979036054a052b2a92c53 I0704 18:15:22.337279 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 24.460849ms to process 1 records I0704 18:15:22.337453 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=48ff60bce47b43e9b8020379d2d2b607992ce31d3b9052c4fff56a2702dbf7da I0704 18:15:22.337486 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=f25a6b23ffab7e2a3418203bdf0a90bf7a7d596870aae3666fbf2bf888adf6e7 I0704 18:15:22.337496 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 37.299409ms to process 2 records I0704 18:15:22.337508 1 tasks_processing.go:74] worker 25 stopped. I0704 18:15:22.338056 1 prometheus_rules.go:88] Prometheus rules successfully created I0704 18:15:22.338178 1 controller.go:489] The operator is still being initialized I0704 18:15:22.338234 1 controller.go:512] The operator is healthy I0704 18:15:22.338401 1 tasks_processing.go:74] worker 12 stopped. I0704 18:15:22.339507 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=0ed2a457c8c3bb8d28a5c656168432612c563bdf5f5dd4498991d6535f128645 I0704 18:15:22.339749 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0704 18:15:22.340097 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0704 18:15:22.340123 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0704 18:15:22.340403 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=25384bcac19eacfb63c64bd569f534b7cb8b47be13baa1f4db2ad4b862a30afd I0704 18:15:22.340427 1 gather.go:177] gatherer "clusterconfig" function "crds" took 50.095419ms to process 2 records I0704 18:15:22.342468 1 tasks_processing.go:74] worker 1 stopped. I0704 18:15:22.342487 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 64.602623ms to process 0 records W0704 18:15:22.342786 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0704 18:15:22.344357 1 base_controller.go:82] Caches are synced for ConfigController I0704 18:15:22.344371 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0704 18:15:22.344777 1 tasks_processing.go:74] worker 28 stopped. I0704 18:15:22.344789 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 67.97039ms to process 0 records E0704 18:15:22.349611 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27d058ac29-d71d-4a05-b468-4df841f89a42%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:51510->172.30.0.10:53: read: connection refused I0704 18:15:22.349633 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27d058ac29-d71d-4a05-b468-4df841f89a42%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:51510->172.30.0.10:53: read: connection refused I0704 18:15:22.349663 1 tasks_processing.go:74] worker 26 stopped. I0704 18:15:22.350006 1 recorder.go:75] Recording config/version with fingerprint=d93eeb0c37d2e0edbb9286ad962f0d888de85c3816ed669ec94877e440ba8d4f I0704 18:15:22.350023 1 recorder.go:75] Recording config/id with fingerprint=326c4133d36947ac2d6b0393ebacab645d91d01ce8be563367cd3190ef3ce681 I0704 18:15:22.350030 1 gather.go:177] gatherer "clusterconfig" function "version" took 71.855ms to process 2 records I0704 18:15:22.352211 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 18:15:22.367355 1 tasks_processing.go:74] worker 14 stopped. E0704 18:15:22.367382 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0704 18:15:22.367391 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0704 18:15:22.367396 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0704 18:15:22.367415 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0704 18:15:22.367470 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=f73fcd48d6f9ebbc83253a1b5f23ef59d0683465ec46e474389103c717467dfb I0704 18:15:22.367483 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0704 18:15:22.367489 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0704 18:15:22.367496 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0704 18:15:22.367535 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0704 18:15:22.367544 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0704 18:15:22.367550 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 68.369334ms to process 7 records I0704 18:15:22.367635 1 tasks_processing.go:74] worker 21 stopped. I0704 18:15:22.368966 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-k728t with fingerprint=73f6b1fb54b8e4b08eb1b0fdebc785ad9bb86efb912be3172f85514d8cc45754 I0704 18:15:22.369023 1 recorder.go:75] Recording config/running_containers with fingerprint=df5058e71e4cdb56108abb0eef62620af5e18980e17944ba3345c668a74bf5f9 I0704 18:15:22.369034 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 54.215767ms to process 2 records I0704 18:15:22.397375 1 tasks_processing.go:74] worker 13 stopped. I0704 18:15:22.397401 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 99.476308ms to process 0 records I0704 18:15:22.400102 1 tasks_processing.go:74] worker 19 stopped. I0704 18:15:22.400140 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0704 18:15:22.400152 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 102.273148ms to process 1 records I0704 18:15:22.402001 1 tasks_processing.go:74] worker 30 stopped. I0704 18:15:22.402031 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0704 18:15:22.402042 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 123.952856ms to process 1 records I0704 18:15:22.402065 1 tasks_processing.go:74] worker 3 stopped. I0704 18:15:22.402081 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 93.482986ms to process 0 records I0704 18:15:22.412340 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0704 18:15:22.416402 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:46528->172.30.0.10:53: read: connection refused I0704 18:15:22.416422 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:46528->172.30.0.10:53: read: connection refused I0704 18:15:22.417423 1 base_controller.go:82] Caches are synced for LoggingSyncer I0704 18:15:22.417441 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0704 18:15:22.450618 1 tasks_processing.go:74] worker 20 stopped. E0704 18:15:22.450640 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0704 18:15:22.450647 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2rbfmdmrbpkc3u3tpkjbhr6nctpsmmt1-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2rbfmdmrbpkc3u3tpkjbhr6nctpsmmt1-primary-cert-bundle-secret" not found I0704 18:15:22.450731 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=5d7e142f26e4f4ee7f5c66b5a564b2708de3887ac318683fca5f1bf919e4aed8 I0704 18:15:22.450746 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 138.7133ms to process 1 records W0704 18:15:23.337989 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0704 18:15:23.529759 1 gather_cluster_operator_pods_and_events.go:121] Found 18 pods with 21 containers I0704 18:15:23.529777 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1198372 bytes I0704 18:15:23.530377 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-5dvsp pod in namespace openshift-dns (previous: false). I0704 18:15:23.760144 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-5dvsp pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-5dvsp\" is waiting to start: ContainerCreating" I0704 18:15:23.760167 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-5dvsp\" is waiting to start: ContainerCreating" I0704 18:15:23.760176 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-5dvsp pod in namespace openshift-dns (previous: false). I0704 18:15:23.935250 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-5dvsp pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-5dvsp\" is waiting to start: ContainerCreating" I0704 18:15:23.935272 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-5dvsp\" is waiting to start: ContainerCreating" I0704 18:15:23.935315 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-w9wz6 pod in namespace openshift-dns (previous: false). I0704 18:15:23.953325 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0704 18:15:24.156060 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-w9wz6 pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-w9wz6\" is waiting to start: ContainerCreating" I0704 18:15:24.156080 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-w9wz6\" is waiting to start: ContainerCreating" I0704 18:15:24.156088 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-w9wz6 pod in namespace openshift-dns (previous: false). I0704 18:15:24.332643 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-w9wz6 pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-w9wz6\" is waiting to start: ContainerCreating" I0704 18:15:24.332663 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-w9wz6\" is waiting to start: ContainerCreating" I0704 18:15:24.332673 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-d6g4b pod in namespace openshift-dns (previous: false). W0704 18:15:24.338258 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0704 18:15:24.354400 1 tasks_processing.go:74] worker 4 stopped. I0704 18:15:24.354454 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=38acfa7f63cd8ea288a669e920d57f745980566383bf1beb969a79a31c451eaf I0704 18:15:24.354484 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=be8418cd809dd2608f83004a7b45c7d839b6057b369d93ee4369c55e4e0aff5f I0704 18:15:24.354514 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0704 18:15:24.354541 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=c56ccd02b2896362820c57d79add2db26e31c71562f9e0d6864276af0e846c72 I0704 18:15:24.354558 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0704 18:15:24.354596 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=e4c68ee9c2c0e71f79e6dcd8e1e5e308a22e6821b0e93679f2d18be26fa7d6f6 I0704 18:15:24.354629 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=cb23940c5856a505b50e60c41cb7b56c2b90b3354f8b037d8094903aa06eabe4 I0704 18:15:24.354655 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=f5f03042bb325ee4350f529d3473e71c6e297780a8ab5492857c8b17272d83a9 I0704 18:15:24.354685 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=2f4fe790e88977140e51eace3304c18523973532e0a9d2cf0f07b9fa616d0bd9 I0704 18:15:24.354696 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=e5ff11d57817f84a678f6fa9565af55bd1120227c16a21933637ab62675a6d70 I0704 18:15:24.354713 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=9d6081cb2e5001074dbce28e49c0bf0539b4fe78148671698b10e6a7ce235185 I0704 18:15:24.354724 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0704 18:15:24.354740 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=baad5421de71b0446dfde7d7037f118e178e790100c6038cdea0182e5f848589 I0704 18:15:24.354750 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0704 18:15:24.354766 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=b6cb3b21d85419b21020bbc172e2fd23a90e18746ad99f220ba7e819a15e9a1b I0704 18:15:24.354776 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0704 18:15:24.354797 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=4ea9535b6192f9ba40f5a08e993e5b390d2167db38070dbdee0af2827dcafdda I0704 18:15:24.354819 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0704 18:15:24.354853 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=bf61b6157b84f49852301bc02191d3537ebe2f77eb37f503a0edb2807ceff404 I0704 18:15:24.354995 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=6b20e6a36a2e1748a333c771e5233227105d67d0a40ae89305c377e783c20d58 I0704 18:15:24.355006 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0704 18:15:24.355014 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0704 18:15:24.355038 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0704 18:15:24.355060 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=89ed9d8d343c36aee23eda7565b94f83f5f321949199b8b3d269428f7e3d8e2a I0704 18:15:24.355086 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=81f901aba906d86362a5b29643665ec4fb64f88b4b1594158e632d888361c389 I0704 18:15:24.355096 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0704 18:15:24.355111 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=79d9d9577a296c4f8dd12fdb15b176a0340324de612ddb4c9230041d4b22f604 I0704 18:15:24.355121 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0704 18:15:24.355133 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=5ddcf4fa26222f69571f403c938aef950a51cbbdb177fd8fa5c142548f35b472 I0704 18:15:24.355148 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=35e3a0954b1a0af5d1d97acc0be11c703d9bffc2dd59f99dc8b796c91e62445d I0704 18:15:24.355162 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=be780ee544996044d44b894dbccdffab5fad446a042ab6a83ee9f7554dd7fdb0 I0704 18:15:24.355178 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=669b785e96ca940e5117de1ab4fe8707a46669e39a9b95069c66f56a873ac01c I0704 18:15:24.355191 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=45374b7098681818975dd5b7e06799900052b508f2195899bc28c0641656be9d I0704 18:15:24.355221 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=188cce7f33faf8d77aae3706aa1cdf76001a03a648cf2f9b97eb978c2a8e58ce I0704 18:15:24.355236 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0704 18:15:24.355245 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0704 18:15:24.355252 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.054507889s to process 36 records I0704 18:15:24.532299 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 18:15:24.532323 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-hfwfw pod in namespace openshift-dns (previous: false). I0704 18:15:24.753558 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 18:15:24.753581 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-hgjpw pod in namespace openshift-dns (previous: false). I0704 18:15:24.932511 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 18:15:24.932584 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-67b7c5fbc9-qbztv pod in namespace openshift-image-registry (previous: false). I0704 18:15:25.132565 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-67b7c5fbc9-qbztv pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-67b7c5fbc9-qbztv\" is waiting to start: ContainerCreating" I0704 18:15:25.132586 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-67b7c5fbc9-qbztv\" is waiting to start: ContainerCreating" I0704 18:15:25.132629 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-67b7c5fbc9-s9fr9 pod in namespace openshift-image-registry (previous: false). I0704 18:15:25.325433 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 18:15:25.325492 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-8f649fcb4-2rmxg pod in namespace openshift-image-registry (previous: false). W0704 18:15:25.338343 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0704 18:15:25.535397 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-8f649fcb4-2rmxg pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-8f649fcb4-2rmxg\" is waiting to start: ContainerCreating" I0704 18:15:25.535417 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-8f649fcb4-2rmxg\" is waiting to start: ContainerCreating" I0704 18:15:25.535427 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-9bvnt pod in namespace openshift-image-registry (previous: false). I0704 18:15:25.732754 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 18:15:25.732785 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-c9674 pod in namespace openshift-image-registry (previous: false). I0704 18:15:25.932402 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 18:15:25.932424 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-kx722 pod in namespace openshift-image-registry (previous: false). I0704 18:15:26.131353 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 18:15:26.131380 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-588f4bbf99-x2rwk pod in namespace openshift-ingress (previous: false). I0704 18:15:26.333523 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-588f4bbf99-x2rwk pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-588f4bbf99-x2rwk\" is waiting to start: ContainerCreating" I0704 18:15:26.333548 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-588f4bbf99-x2rwk\" is waiting to start: ContainerCreating" I0704 18:15:26.333562 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6444bc7b68-2qrxx pod in namespace openshift-ingress (previous: false). W0704 18:15:26.338425 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0704 18:15:26.531306 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-6444bc7b68-2qrxx pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-6444bc7b68-2qrxx\" is waiting to start: ContainerCreating" I0704 18:15:26.531328 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-6444bc7b68-2qrxx\" is waiting to start: ContainerCreating" I0704 18:15:26.531340 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6444bc7b68-zzrcq pod in namespace openshift-ingress (previous: false). I0704 18:15:26.732967 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-6444bc7b68-zzrcq pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-6444bc7b68-zzrcq\" is waiting to start: ContainerCreating" I0704 18:15:26.732989 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-6444bc7b68-zzrcq\" is waiting to start: ContainerCreating" I0704 18:15:26.733029 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-fpv6l pod in namespace openshift-ingress-canary (previous: false). I0704 18:15:26.931987 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-fpv6l pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-fpv6l\" is waiting to start: ContainerCreating" I0704 18:15:26.932008 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-fpv6l\" is waiting to start: ContainerCreating" I0704 18:15:26.932019 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-lz8jm pod in namespace openshift-ingress-canary (previous: false). I0704 18:15:27.133169 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-lz8jm pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-lz8jm\" is waiting to start: ContainerCreating" I0704 18:15:27.133196 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-lz8jm\" is waiting to start: ContainerCreating" I0704 18:15:27.133214 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for migrator container migrator-7d5f866c57-grg97 pod in namespace openshift-kube-storage-version-migrator (previous: false). W0704 18:15:27.333902 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0704 18:15:27.333944 1 tasks_processing.go:74] worker 17 stopped. E0704 18:15:27.333958 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0704 18:15:27.333972 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0704 18:15:27.333992 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0704 18:15:27.334005 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.036247502s to process 1 records I0704 18:15:27.335025 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for graceful-termination container migrator-7d5f866c57-grg97 pod in namespace openshift-kube-storage-version-migrator (previous: false). I0704 18:15:27.536895 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-storage-version-migrator-operator container kube-storage-version-migrator-operator-74848b4cb9-ds9g4 pod in namespace openshift-kube-storage-version-migrator-operator (previous: false). I0704 18:15:27.735034 1 tasks_processing.go:74] worker 23 stopped. I0704 18:15:27.735259 1 recorder.go:75] Recording events/openshift-dns with fingerprint=5c688230dd56488554d4971c51ef2da227b62eae84a53239797aecb4ea8bd49e I0704 18:15:27.735439 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=3e2460f15596b1c0924cfc7212cfade0fccf6d6484499b22b1381c890917f5ae I0704 18:15:27.735501 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=353c4559712643eaa92075daa378b6b3dade694833d849a51b4f5191d879fd25 I0704 18:15:27.735597 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=d262ca232fecfdc6cc2cdcb7c91e42f1eccfde8f18ec7b0374bc15c0c069f224 I0704 18:15:27.735636 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=51b26cf5d3d8d9315f14e4edf73bc0d079efd6d60c03c8a6acaf9b9f2e0f3b69 I0704 18:15:27.735674 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator with fingerprint=0fedf594eda0c44d2156e9b66a535fa6aaa0c1d650c0e6d0139bf83b9452467a I0704 18:15:27.735772 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator-operator with fingerprint=83ef3e97b6fdac3db7b70730c601b7fb10706f24d5f03d4b918010d9b66757e7 I0704 18:15:27.736039 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-w9wz6 with fingerprint=2baf32c2162f99a8d4cfeb75a0b8c4777db917173870b7230550f0272108bf7a I0704 18:15:27.736269 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-67b7c5fbc9-qbztv with fingerprint=be23ff7e0018c93bc45cf5d11b0711cc7e0c5e511946cc6e77a3c7cadabce7be I0704 18:15:27.736424 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-67b7c5fbc9-s9fr9 with fingerprint=ac2b4326e323b3c06b46c8cd5e5b19c06a2ed79988222657e6fc65dc697bfa12 I0704 18:15:27.736600 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-8f649fcb4-2rmxg with fingerprint=8be6420ea1095b5e6ed3aacd1cf2b8d5029d003b0c2936ce21b51cbf91e657fa I0704 18:15:27.736723 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-fpv6l with fingerprint=fa93d07c8aeda09c5f6f3129debc12f20057e1254b0afebf576fcdadad58c02c I0704 18:15:27.736763 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-grg97/migrator_current.log with fingerprint=92a44466bef779107668b3da1c45b535140d827a190eccc8bcb12dd1854046c0 I0704 18:15:27.736789 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-grg97/graceful-termination_current.log with fingerprint=7480d077145d187c9d347a9b9f117e2dbd500da4ca01afa64ef266fc9d70b4ec I0704 18:15:27.736960 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator-operator/logs/kube-storage-version-migrator-operator-74848b4cb9-ds9g4/kube-storage-version-migrator-operator_current.log with fingerprint=3858a0d741434c5bd0f58c1351852414c417e400ea268e57e34bc85c1a73818d I0704 18:15:27.736999 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 5.452153002s to process 15 records I0704 18:15:34.865279 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 18:15:35.206706 1 tasks_processing.go:74] worker 5 stopped. I0704 18:15:35.206748 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0704 18:15:35.206764 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.907629886s to process 1 records I0704 18:15:35.884004 1 tasks_processing.go:74] worker 16 stopped. I0704 18:15:35.884411 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=9d475996b970b5499163b99cb7866431c26bc05ed77261e8b402c7515458063c I0704 18:15:35.884436 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.60562727s to process 1 records E0704 18:15:35.884521 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.607s with: function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0704 18:15:35.885638 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machines" failed with an error, function "support_secret" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0704 18:15:35.885658 1 periodic.go:209] Running workloads gatherer I0704 18:15:35.885687 1 tasks_processing.go:45] number of workers: 2 I0704 18:15:35.885702 1 tasks_processing.go:69] worker 1 listening for tasks. I0704 18:15:35.885709 1 tasks_processing.go:71] worker 1 working on workload_info task. I0704 18:15:35.885715 1 tasks_processing.go:69] worker 0 listening for tasks. I0704 18:15:35.885729 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0704 18:15:35.914185 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 22s for image data I0704 18:15:35.915008 1 tasks_processing.go:74] worker 0 stopped. I0704 18:15:35.915026 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 29.262608ms to process 0 records I0704 18:15:35.922515 1 gather_workloads_info.go:387] No image sha256:80748ba08e1c264a8c105e7f607eff386a66378e024443a844993ee9292858c1 (10ms) I0704 18:15:35.932057 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (10ms) I0704 18:15:35.941710 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (10ms) I0704 18:15:35.950914 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (9ms) I0704 18:15:35.959932 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (9ms) I0704 18:15:35.969081 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (9ms) I0704 18:15:35.978329 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (9ms) I0704 18:15:35.987475 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (9ms) I0704 18:15:35.999057 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (12ms) I0704 18:15:36.008174 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (9ms) I0704 18:15:36.025859 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (18ms) I0704 18:15:36.123267 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (97ms) I0704 18:15:36.222301 1 gather_workloads_info.go:387] No image sha256:fac526ab9335a9a8decb028a30673f4ae718deae8da114f36103b6438ba6458f (99ms) I0704 18:15:36.323009 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (101ms) I0704 18:15:36.423451 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (100ms) I0704 18:15:36.523844 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0704 18:15:36.623352 1 gather_workloads_info.go:387] No image sha256:95418d0ba6fc0aa96647d97cf6530967f7d7ae97a82ce7f7e41c8e44310b0b72 (99ms) I0704 18:15:36.625670 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 18:15:36.723726 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (100ms) I0704 18:15:36.823586 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (100ms) I0704 18:15:36.824319 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 18:15:36.933644 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (110ms) I0704 18:15:37.029474 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (96ms) I0704 18:15:37.126998 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (97ms) I0704 18:15:37.223490 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (96ms) I0704 18:15:37.323389 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (100ms) I0704 18:15:37.423100 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (100ms) I0704 18:15:37.522661 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0704 18:15:37.623026 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0704 18:15:37.723564 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (101ms) I0704 18:15:37.823916 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (100ms) I0704 18:15:37.924753 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (101ms) I0704 18:15:38.024112 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (99ms) I0704 18:15:38.122627 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (98ms) I0704 18:15:38.223213 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (101ms) I0704 18:15:38.323017 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0704 18:15:38.423554 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (101ms) I0704 18:15:38.423588 1 tasks_processing.go:74] worker 1 stopped. E0704 18:15:38.423599 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0704 18:15:38.423978 1 recorder.go:75] Recording config/workload_info with fingerprint=8e2b2f6dad39c0c493e8dd266d6b5d8198441436ae61a212ea38babfbf3c0851 I0704 18:15:38.423995 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.53787092s to process 1 records E0704 18:15:38.424020 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.538s with: function \"workload_info\" failed with an error" I0704 18:15:38.425125 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0704 18:15:38.425140 1 periodic.go:209] Running conditional gatherer I0704 18:15:38.432102 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0704 18:15:38.439062 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.12:40003->172.30.0.10:53: read: connection refused E0704 18:15:38.439338 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 18:15:38.439415 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0704 18:15:38.448324 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0704 18:15:38.448343 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 18:15:38.448349 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 18:15:38.448352 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 18:15:38.448355 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 18:15:38.448358 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 18:15:38.448361 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 18:15:38.448363 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 18:15:38.448366 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 18:15:38.448369 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0704 18:15:38.448384 1 tasks_processing.go:45] number of workers: 3 I0704 18:15:38.448391 1 tasks_processing.go:69] worker 2 listening for tasks. I0704 18:15:38.448395 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0704 18:15:38.448460 1 tasks_processing.go:69] worker 1 listening for tasks. I0704 18:15:38.448475 1 tasks_processing.go:71] worker 1 working on remote_configuration task. I0704 18:15:38.448481 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0704 18:15:38.448481 1 tasks_processing.go:69] worker 0 listening for tasks. I0704 18:15:38.448489 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0704 18:15:38.448491 1 tasks_processing.go:74] worker 0 stopped. I0704 18:15:38.448499 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 925ns to process 1 records I0704 18:15:38.448532 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0704 18:15:38.448540 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.274µs to process 1 records I0704 18:15:38.448545 1 tasks_processing.go:74] worker 1 stopped. I0704 18:15:38.448745 1 tasks_processing.go:74] worker 2 stopped. I0704 18:15:38.448756 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 258.601µs to process 0 records I0704 18:15:38.448779 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.12:40003->172.30.0.10:53: read: connection refused I0704 18:15:38.448797 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0704 18:15:38.474772 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=c57c180b09550571fa60ecf4b7f189cd30d36b87d8b93a79f5aeba1a8b1b24b4 I0704 18:15:38.474947 1 diskrecorder.go:70] Writing 110 records to /var/lib/insights-operator/insights-2026-07-04-181538.tar.gz I0704 18:15:38.482147 1 diskrecorder.go:51] Wrote 110 records to disk in 7ms I0704 18:15:38.482197 1 periodic.go:278] Gathering cluster info every 2h0m0s I0704 18:15:38.482215 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0704 18:15:51.894414 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 18:16:41.453904 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="4204c2a06c42363407fbaa7924941d79cc5b18fff713436f5c0644683613bccb") W0704 18:16:41.453944 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0704 18:16:41.454022 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0704 18:16:41.454033 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="922e3186be75f9eff41ca45b1d0e77936578ce8fed0666c1dabc3f3812105941")