W0704 06:20:29.154083 1 cmd.go:257] Using insecure, self-signed certificates I0704 06:20:30.218668 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 06:20:30.218988 1 observer_polling.go:159] Starting file observer I0704 06:20:30.889018 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0704 06:20:30.889228 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0704 06:20:30.889807 1 secure_serving.go:57] Forcing use of http/1.1 only W0704 06:20:30.889833 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. I0704 06:20:30.889834 1 simple_featuregate_reader.go:171] Starting feature-gate-detector W0704 06:20:30.889839 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0704 06:20:30.889871 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0704 06:20:30.889876 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0704 06:20:30.889882 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0704 06:20:30.889887 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0704 06:20:30.896902 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0704 06:20:30.896955 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"242ae091-69eb-4a49-baa2-f655919adfd9", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0704 06:20:30.899940 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0704 06:20:30.899964 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0704 06:20:30.899965 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0704 06:20:30.899998 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0704 06:20:30.899968 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0704 06:20:30.900013 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0704 06:20:30.900196 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-3164530909/tls.crt::/tmp/serving-cert-3164530909/tls.key" I0704 06:20:30.900582 1 secure_serving.go:213] Serving securely on [::]:8443 I0704 06:20:30.900610 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0704 06:20:30.903359 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0704 06:20:30.903384 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0704 06:20:30.903489 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0704 06:20:30.910258 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0704 06:20:30.910275 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0704 06:20:30.914265 1 secretconfigobserver.go:119] support secret does not exist I0704 06:20:30.918428 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0704 06:20:30.923574 1 secretconfigobserver.go:119] support secret does not exist I0704 06:20:30.932748 1 recorder.go:161] Pruning old reports every 4h46m39s, max age is 288h0m0s I0704 06:20:30.938146 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0704 06:20:30.938167 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0704 06:20:30.938170 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0704 06:20:30.938181 1 insightsreport.go:296] Starting report retriever I0704 06:20:30.938184 1 periodic.go:209] Running clusterconfig gatherer I0704 06:20:30.938189 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0704 06:20:30.938236 1 tasks_processing.go:45] number of workers: 32 I0704 06:20:30.938263 1 tasks_processing.go:69] worker 1 listening for tasks. I0704 06:20:30.938277 1 tasks_processing.go:69] worker 14 listening for tasks. I0704 06:20:30.938278 1 tasks_processing.go:69] worker 0 listening for tasks. I0704 06:20:30.938283 1 tasks_processing.go:71] worker 14 working on oauths task. I0704 06:20:30.938279 1 tasks_processing.go:71] worker 1 working on pdbs task. I0704 06:20:30.938289 1 tasks_processing.go:69] worker 15 listening for tasks. I0704 06:20:30.938295 1 tasks_processing.go:69] worker 7 listening for tasks. I0704 06:20:30.938301 1 tasks_processing.go:69] worker 24 listening for tasks. I0704 06:20:30.938308 1 tasks_processing.go:69] worker 2 listening for tasks. I0704 06:20:30.938310 1 tasks_processing.go:69] worker 25 listening for tasks. I0704 06:20:30.938309 1 tasks_processing.go:69] worker 23 listening for tasks. I0704 06:20:30.938316 1 tasks_processing.go:69] worker 5 listening for tasks. I0704 06:20:30.938318 1 tasks_processing.go:69] worker 16 listening for tasks. I0704 06:20:30.938322 1 tasks_processing.go:69] worker 26 listening for tasks. I0704 06:20:30.938325 1 tasks_processing.go:69] worker 6 listening for tasks. I0704 06:20:30.938319 1 tasks_processing.go:69] worker 4 listening for tasks. I0704 06:20:30.938327 1 tasks_processing.go:69] worker 3 listening for tasks. I0704 06:20:30.938330 1 tasks_processing.go:69] worker 17 listening for tasks. I0704 06:20:30.938341 1 tasks_processing.go:69] worker 29 listening for tasks. I0704 06:20:30.938334 1 tasks_processing.go:69] worker 27 listening for tasks. I0704 06:20:30.938337 1 tasks_processing.go:69] worker 18 listening for tasks. I0704 06:20:30.938342 1 tasks_processing.go:69] worker 30 listening for tasks. I0704 06:20:30.938350 1 tasks_processing.go:69] worker 31 listening for tasks. I0704 06:20:30.938353 1 tasks_processing.go:69] worker 22 listening for tasks. I0704 06:20:30.938344 1 tasks_processing.go:69] worker 19 listening for tasks. I0704 06:20:30.938357 1 tasks_processing.go:69] worker 28 listening for tasks. I0704 06:20:30.938360 1 tasks_processing.go:69] worker 10 listening for tasks. I0704 06:20:30.938351 1 tasks_processing.go:69] worker 21 listening for tasks. I0704 06:20:30.938357 1 tasks_processing.go:69] worker 20 listening for tasks. I0704 06:20:30.938365 1 tasks_processing.go:69] worker 8 listening for tasks. I0704 06:20:30.938368 1 tasks_processing.go:69] worker 9 listening for tasks. I0704 06:20:30.938370 1 tasks_processing.go:69] worker 12 listening for tasks. I0704 06:20:30.938373 1 tasks_processing.go:69] worker 11 listening for tasks. I0704 06:20:30.938376 1 tasks_processing.go:69] worker 13 listening for tasks. I0704 06:20:30.938445 1 tasks_processing.go:71] worker 13 working on mutating_webhook_configurations task. I0704 06:20:30.938467 1 tasks_processing.go:71] worker 16 working on config_maps task. I0704 06:20:30.938471 1 tasks_processing.go:71] worker 15 working on version task. I0704 06:20:30.938473 1 tasks_processing.go:71] worker 26 working on olm_operators task. I0704 06:20:30.938478 1 tasks_processing.go:71] worker 28 working on openshift_logging task. I0704 06:20:30.938528 1 tasks_processing.go:71] worker 7 working on overlapping_namespace_uids task. I0704 06:20:30.938467 1 tasks_processing.go:71] worker 5 working on metrics task. I0704 06:20:30.938542 1 tasks_processing.go:71] worker 2 working on silenced_alerts task. W0704 06:20:30.938759 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 06:20:30.938819 1 tasks_processing.go:71] worker 2 working on image_registries task. I0704 06:20:30.938842 1 tasks_processing.go:71] worker 8 working on support_secret task. I0704 06:20:30.939005 1 tasks_processing.go:71] worker 27 working on ingress_certificates task. I0704 06:20:30.939246 1 tasks_processing.go:71] worker 22 working on crds task. I0704 06:20:30.939335 1 tasks_processing.go:71] worker 10 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0704 06:20:30.939386 1 tasks_processing.go:71] worker 19 working on sap_pods task. I0704 06:20:30.939479 1 tasks_processing.go:71] worker 9 working on pod_network_connectivity_checks task. I0704 06:20:30.939556 1 tasks_processing.go:71] worker 18 working on cluster_apiserver task. I0704 06:20:30.939700 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 167.453µs to process 0 records I0704 06:20:30.938454 1 tasks_processing.go:71] worker 4 working on infrastructures task. I0704 06:20:30.939869 1 tasks_processing.go:71] worker 30 working on active_alerts task. W0704 06:20:30.939905 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 06:20:30.939918 1 tasks_processing.go:71] worker 30 working on storage_classes task. I0704 06:20:30.939930 1 tasks_processing.go:71] worker 12 working on proxies task. I0704 06:20:30.938770 1 tasks_processing.go:71] worker 25 working on sap_datahubs task. I0704 06:20:30.940138 1 tasks_processing.go:71] worker 21 working on aggregated_monitoring_cr_names task. I0704 06:20:30.938829 1 tasks_processing.go:71] worker 24 working on openstack_dataplanedeployments task. I0704 06:20:30.940305 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 35.678µs to process 0 records I0704 06:20:30.939406 1 tasks_processing.go:71] worker 11 working on dvo_metrics task. I0704 06:20:30.940386 1 tasks_processing.go:71] worker 20 working on clusterroles task. I0704 06:20:30.938457 1 tasks_processing.go:71] worker 3 working on image task. I0704 06:20:30.938462 1 tasks_processing.go:71] worker 6 working on feature_gates task. I0704 06:20:30.938450 1 tasks_processing.go:71] worker 23 working on install_plans task. I0704 06:20:30.938449 1 tasks_processing.go:71] worker 29 working on container_runtime_configs task. W0704 06:20:30.938778 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 06:20:30.940988 1 tasks_processing.go:71] worker 5 working on image_pruners task. I0704 06:20:30.940298 1 tasks_processing.go:71] worker 31 working on validating_webhook_configurations task. I0704 06:20:30.941165 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 2.439674ms to process 0 records I0704 06:20:30.938461 1 tasks_processing.go:71] worker 17 working on lokistack task. I0704 06:20:30.938466 1 tasks_processing.go:71] worker 0 working on openshift_machine_api_events task. I0704 06:20:30.943031 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0704 06:20:30.943051 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0704 06:20:30.943057 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0704 06:20:30.943061 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0704 06:20:30.943080 1 controller.go:489] The operator is still being initialized I0704 06:20:30.943087 1 controller.go:512] The operator is healthy I0704 06:20:30.956115 1 tasks_processing.go:71] worker 25 working on openstack_dataplanenodesets task. I0704 06:20:30.956131 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 16.052631ms to process 0 records I0704 06:20:30.956151 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 16.68529ms to process 0 records I0704 06:20:30.956163 1 tasks_processing.go:71] worker 19 working on monitoring_persistent_volumes task. I0704 06:20:30.956371 1 tasks_processing.go:71] worker 14 working on node_logs task. I0704 06:20:30.956621 1 recorder.go:75] Recording config/oauth with fingerprint=b6dd31e27102d6e356b7d1ed57105e8b901943598b387d621bc23ee60cd05c65 I0704 06:20:30.956635 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 18.071901ms to process 1 records I0704 06:20:30.961037 1 tasks_processing.go:71] worker 9 working on nodenetworkstates task. E0704 06:20:30.961047 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0704 06:20:30.961059 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 21.539966ms to process 0 records I0704 06:20:30.961492 1 tasks_processing.go:71] worker 24 working on machine_sets task. I0704 06:20:30.961508 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 21.178042ms to process 0 records I0704 06:20:30.962979 1 tasks_processing.go:71] worker 17 working on authentication task. I0704 06:20:30.962988 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 21.481539ms to process 0 records I0704 06:20:30.972717 1 tasks_processing.go:71] worker 29 working on tsdb_status task. I0704 06:20:30.972737 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 31.782305ms to process 0 records W0704 06:20:30.972746 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 06:20:30.972753 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 34.241742ms to process 0 records I0704 06:20:30.972760 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 26.91µs to process 0 records I0704 06:20:30.972769 1 tasks_processing.go:71] worker 29 working on machine_config_pools task. I0704 06:20:30.972816 1 tasks_processing.go:71] worker 28 working on sap_config task. I0704 06:20:30.982730 1 tasks_processing.go:71] worker 3 working on ingress task. I0704 06:20:30.982852 1 recorder.go:75] Recording config/image with fingerprint=2d59f477cf9dfa9bfab7d5dd7fb9c52f38bdafc5ed20fecbeafdc756ee2eb73f I0704 06:20:30.982869 1 gather.go:177] gatherer "clusterconfig" function "image" took 42.138574ms to process 1 records I0704 06:20:31.000584 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0704 06:20:31.000592 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0704 06:20:31.000608 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0704 06:20:31.003763 1 base_controller.go:82] Caches are synced for ConfigController I0704 06:20:31.003776 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0704 06:20:31.012376 1 tasks_processing.go:71] worker 4 working on service_accounts task. I0704 06:20:31.013117 1 recorder.go:75] Recording config/infrastructure with fingerprint=9974a238aa9eb8c5484630e7dd6c48032cb02811aea564d35949d335447ac0c4 I0704 06:20:31.013136 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 72.644796ms to process 1 records E0704 06:20:31.013148 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0704 06:20:31.013154 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 73.590614ms to process 0 records I0704 06:20:31.013158 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 70.812529ms to process 0 records I0704 06:20:31.013197 1 recorder.go:75] Recording config/proxy with fingerprint=49d51c539c2f5262e4e074a9c51138b3e475643f41091cf8495fc99371095707 I0704 06:20:31.013205 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 72.687613ms to process 1 records I0704 06:20:31.013212 1 tasks_processing.go:71] worker 12 working on storage_cluster task. I0704 06:20:31.013284 1 tasks_processing.go:71] worker 0 working on networks task. I0704 06:20:31.013292 1 tasks_processing.go:71] worker 8 working on machine_autoscalers task. I0704 06:20:31.015930 1 tasks_processing.go:71] worker 28 working on ceph_cluster task. I0704 06:20:31.015952 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 43.099161ms to process 0 records I0704 06:20:31.016221 1 tasks_processing.go:71] worker 24 working on certificate_signing_requests task. I0704 06:20:31.016274 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 54.71866ms to process 0 records I0704 06:20:31.017811 1 tasks_processing.go:71] worker 1 working on machine_healthchecks task. I0704 06:20:31.018096 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=eb3ccb363a614f4da529ebcfd70b5637e559c89ac5b132ee23f7f11bbb297f74 I0704 06:20:31.018144 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=845f4e8a657bb04630bec02ce72e23e8e207fda9d24748a16c65f25c0cf64f25 I0704 06:20:31.018171 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=cc7d8fd15dc78a09ecb9dfaf418ff8378cfdc6d3bf2754bed563e3f9795275af I0704 06:20:31.018244 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 79.515756ms to process 3 records I0704 06:20:31.018609 1 tasks_processing.go:71] worker 2 working on qemu_kubevirt_launcher_logs task. I0704 06:20:31.019147 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=b88965c1b3d46e91035572be5bffdc3843df85db34382150c7dd54de9ff87133 I0704 06:20:31.019162 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 79.758623ms to process 1 records I0704 06:20:31.019619 1 tasks_processing.go:71] worker 9 working on nodenetworkconfigurationpolicies task. I0704 06:20:31.019632 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 58.569852ms to process 0 records I0704 06:20:31.019644 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 63.506633ms to process 0 records I0704 06:20:31.019746 1 tasks_processing.go:71] worker 25 working on container_images task. I0704 06:20:31.020243 1 tasks_processing.go:71] worker 6 working on operators task. I0704 06:20:31.021261 1 recorder.go:75] Recording config/featuregate with fingerprint=d930b63a05e5b5ed87d1ca0c8c91dc954d9fe2842dcec202328999f3fdb52c11 I0704 06:20:31.021342 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 79.495572ms to process 1 records I0704 06:20:31.021709 1 recorder.go:75] Recording config/apiserver with fingerprint=ec8c89e27c17871538c68e074c58e6aa2fdda0946813c6e56634cd6881a4449f I0704 06:20:31.021763 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 80.792528ms to process 1 records I0704 06:20:31.021796 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 82.310456ms to process 0 records I0704 06:20:31.021936 1 tasks_processing.go:71] worker 18 working on jaegers task. I0704 06:20:31.022074 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=113debe3660a3c4e8d34dff964dbabadb82c2b311c11bf0f4675c6ac64c0f6a6 I0704 06:20:31.022131 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=3513e8b0dc89d20517a434655ff7ee0456984d5ba3924f9bb5eb28e4b8877df8 I0704 06:20:31.022157 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 81.211935ms to process 2 records I0704 06:20:31.022222 1 tasks_processing.go:71] worker 30 working on operators_pods_and_events task. W0704 06:20:31.022431 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0704 06:20:31.022491 1 tasks_processing.go:71] worker 26 working on schedulers task. I0704 06:20:31.022540 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=7dbc0008e8b69a0368761e7f2224e4ac6c775750e23fcd119f4d5f66705ccb15 I0704 06:20:31.022597 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=3485fbbde65ae49a0f4d3f5d133d22660e65b78bb31427672e0b42279347c330 I0704 06:20:31.022640 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=a9e479758c54faaac035e1cbd82e8d512c995d8b78493045dff7bb9dd632c191 I0704 06:20:31.022671 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 82.954791ms to process 3 records I0704 06:20:31.022794 1 tasks_processing.go:71] worker 13 working on openstack_version task. I0704 06:20:31.023040 1 recorder.go:75] Recording config/authentication with fingerprint=7b5f89708298e6f1dad89a4dbfd8aa9f61aab612c40bbc65575efd7c9deb8a7f I0704 06:20:31.023057 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 58.869607ms to process 1 records I0704 06:20:31.023088 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0704 06:20:31.023087 1 tasks_processing.go:71] worker 17 working on nodes task. I0704 06:20:31.023096 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 83.988538ms to process 1 records I0704 06:20:31.023106 1 tasks_processing.go:71] worker 7 working on openstack_controlplanes task. I0704 06:20:31.023353 1 tasks_processing.go:71] worker 31 working on machine_configs task. I0704 06:20:31.023864 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=52c727170330a5922e2ff90172deb80dd8f10c87d1f769fd7af406f349fd2ad3 I0704 06:20:31.024089 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=974a0358d8a43c1a4598c667ab41a7cf64bdc7f7a9f2e1d6002cb707731deaae I0704 06:20:31.024196 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=946e676340ec7f2028c2724312cd21daf6690abbe8e4827b7af8152881bfe0e6 I0704 06:20:31.024269 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=8c781b207805b35a7c5eb2414e49d1dbc6a82d073ef2fedc2b6201dff6861048 I0704 06:20:31.024346 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=3485b7b5c8044c690da65b943f5460ce334ff861a52af380f906fbe50cdcf045 I0704 06:20:31.024397 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=8c26075ec4fc528f8b1daab90d1569c5741078d019e329571da648b377f0a1db I0704 06:20:31.024467 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=ad60514da2cdf9b75d70e92bb3ca0b3442969b2f8ab96a9fe0a62423e54d25f5 I0704 06:20:31.024568 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0704 06:20:31.024584 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=ed3d150024a188228df14945fe693641c3ef1f8f057a105098adc0f720d05b67 I0704 06:20:31.024617 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=6dbe9c0a14d31dc6dd7940b0c5c6a0b3e32304677428ef4205f2855c067f8eb7 I0704 06:20:31.025243 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0704 06:20:31.025402 1 operator.go:288] started I0704 06:20:31.025414 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=47e1ab35dd87438aae9fc0a5825ef16479918c74f2c0d63ca2dbd8bb7b7c7af8 I0704 06:20:31.025438 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0704 06:20:31.025460 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=e9bde0f7a4f8a9f245d0c2b4e8005a1d8bcb11cf353bd87cffd77132d30ba0f8 I0704 06:20:31.025488 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 82.197189ms to process 11 records I0704 06:20:31.025613 1 tasks_processing.go:71] worker 22 working on cost_management_metrics_configs task. I0704 06:20:31.026103 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=e7afd294a370556c56a8b203994e1af91222e2daad6a72ca6bbd09f3cf0829ea I0704 06:20:31.026342 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=bc32cbd43d1793671da7f13898068ebde4d05c8824fec8952d216e4f05ad1b83 I0704 06:20:31.026353 1 gather.go:177] gatherer "clusterconfig" function "crds" took 86.221534ms to process 2 records I0704 06:20:31.041438 1 tasks_processing.go:71] worker 3 working on machines task. I0704 06:20:31.041628 1 recorder.go:75] Recording config/ingress with fingerprint=24ab5d7c3cce109fa966a64c176f8a7efb7d8f4f9bdfdf563bedba561fe48c73 I0704 06:20:31.041667 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 58.684104ms to process 1 records I0704 06:20:31.042475 1 tasks_processing.go:74] worker 19 stopped. I0704 06:20:31.042503 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 86.291183ms to process 0 records I0704 06:20:31.052054 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 06:20:31.064160 1 tasks_processing.go:74] worker 14 stopped. I0704 06:20:31.064183 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 107.765362ms to process 0 records I0704 06:20:31.112077 1 tasks_processing.go:74] worker 8 stopped. I0704 06:20:31.112141 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 98.680069ms to process 0 records I0704 06:20:31.112165 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 96.150782ms to process 0 records I0704 06:20:31.112171 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 89.003483ms to process 0 records I0704 06:20:31.112179 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 89.327093ms to process 0 records I0704 06:20:31.112183 1 tasks_processing.go:74] worker 28 stopped. I0704 06:20:31.112187 1 tasks_processing.go:74] worker 13 stopped. I0704 06:20:31.112195 1 tasks_processing.go:74] worker 12 stopped. I0704 06:20:31.112191 1 tasks_processing.go:74] worker 7 stopped. I0704 06:20:31.112206 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 98.976885ms to process 0 records I0704 06:20:31.112304 1 tasks_processing.go:74] worker 1 stopped. E0704 06:20:31.112317 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0704 06:20:31.112331 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 94.473529ms to process 0 records I0704 06:20:31.112341 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 86.681083ms to process 0 records I0704 06:20:31.112348 1 tasks_processing.go:74] worker 22 stopped. I0704 06:20:31.112783 1 tasks_processing.go:74] worker 18 stopped. I0704 06:20:31.112795 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 90.778832ms to process 0 records I0704 06:20:31.112810 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 93.155643ms to process 0 records I0704 06:20:31.112817 1 tasks_processing.go:74] worker 9 stopped. I0704 06:20:31.112880 1 tasks_processing.go:74] worker 3 stopped. E0704 06:20:31.112895 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0704 06:20:31.112907 1 gather.go:177] gatherer "clusterconfig" function "machines" took 71.424131ms to process 0 records I0704 06:20:31.119013 1 tasks_processing.go:74] worker 20 stopped. I0704 06:20:31.119190 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=5ef8505862d10cea3a554e3a40ed1f3c934fe6f6a7580321f7bd40be4ccb686c I0704 06:20:31.119284 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=8ac8b761857dde981af58b22ba6ef82306c21baa87c5213ac0d1b273fd243e12 I0704 06:20:31.119296 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 178.609022ms to process 2 records I0704 06:20:31.126151 1 base_controller.go:82] Caches are synced for LoggingSyncer I0704 06:20:31.126165 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0704 06:20:31.136192 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0704 06:20:31.136204 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0704 06:20:31.136207 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0704 06:20:31.136211 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0704 06:20:31.136216 1 controller.go:212] Source scaController *sca.Controller is not ready I0704 06:20:31.136220 1 tasks_processing.go:74] worker 0 stopped. I0704 06:20:31.136237 1 controller.go:489] The operator is still being initialized I0704 06:20:31.136243 1 controller.go:512] The operator is healthy I0704 06:20:31.136371 1 recorder.go:75] Recording config/network with fingerprint=9bd9a0f718b1fbb146ecd21479a62130763eecec54fd9e289a10f55c88869a41 I0704 06:20:31.136390 1 gather.go:177] gatherer "clusterconfig" function "networks" took 122.895051ms to process 1 records I0704 06:20:31.141142 1 tasks_processing.go:74] worker 10 stopped. I0704 06:20:31.141155 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 201.792363ms to process 0 records I0704 06:20:31.155251 1 prometheus_rules.go:88] Prometheus rules successfully created I0704 06:20:31.159985 1 tasks_processing.go:74] worker 24 stopped. I0704 06:20:31.160002 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 143.739896ms to process 0 records E0704 06:20:31.161696 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27b0d80fd4-ef53-4a19-9b43-eaf930202698%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.22:55018->172.30.0.10:53: read: connection refused I0704 06:20:31.161710 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27b0d80fd4-ef53-4a19-9b43-eaf930202698%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.22:55018->172.30.0.10:53: read: connection refused I0704 06:20:31.164501 1 tasks_processing.go:74] worker 5 stopped. I0704 06:20:31.164590 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=bc5d35fd0bb794a01184c09af0fa6b844e72acacabf413815c7a22aa6cd2b0eb I0704 06:20:31.164603 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 223.499476ms to process 1 records I0704 06:20:31.239452 1 tasks_processing.go:74] worker 17 stopped. I0704 06:20:31.239786 1 recorder.go:75] Recording config/node/ip-10-0-0-96.ec2.internal with fingerprint=557acd164ee9fd4f18c6169813fa4bdd5bf633ed5187d4ecaeba0d88d72e5343 I0704 06:20:31.239845 1 recorder.go:75] Recording config/node/ip-10-0-1-82.ec2.internal with fingerprint=7fbc6f82f4a06ae3d28a6ab275e561216adbc0e44c1a23a9f03816f364fcaebd I0704 06:20:31.239893 1 recorder.go:75] Recording config/node/ip-10-0-2-195.ec2.internal with fingerprint=bf67e19dd3a7f9f8aab55dcbe8b970127e6d4011398fcc533764820b7226ef71 I0704 06:20:31.239903 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 216.338678ms to process 3 records I0704 06:20:31.257182 1 gather_logs.go:145] no pods in namespace were found I0704 06:20:31.257200 1 tasks_processing.go:74] worker 2 stopped. I0704 06:20:31.257210 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 238.577216ms to process 0 records I0704 06:20:31.257259 1 tasks_processing.go:74] worker 15 stopped. I0704 06:20:31.257681 1 recorder.go:75] Recording config/version with fingerprint=6f0c482e6ee4c578fdde02522702522e3ccbf22bcf0813c8337f1927cf95e384 I0704 06:20:31.257699 1 recorder.go:75] Recording config/id with fingerprint=d9ec1ceabe5848c81261f9e6de7a064e5c3de98154d53014de6ee846704ef187 I0704 06:20:31.257707 1 gather.go:177] gatherer "clusterconfig" function "version" took 318.77401ms to process 2 records I0704 06:20:31.257771 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=d21e9a13fc526e1dbe40807708278bc6d73a0359f5b85b95ed11dda6d34fa029 I0704 06:20:31.257780 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 235.015404ms to process 1 records I0704 06:20:31.257789 1 tasks_processing.go:74] worker 26 stopped. I0704 06:20:31.278208 1 tasks_processing.go:74] worker 25 stopped. I0704 06:20:31.279560 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-nl6t4 with fingerprint=9690f2de480af297f6e048d9645b050b76ed5641c484f6da3207700859498304 I0704 06:20:31.279639 1 recorder.go:75] Recording config/running_containers with fingerprint=502b338beb72f1e209d48f9f69699e0c2c52b8bff9b84c11b355635288f26a80 I0704 06:20:31.279668 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 258.263395ms to process 2 records I0704 06:20:31.336371 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0704 06:20:31.340154 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.22:44545->172.30.0.10:53: read: connection refused I0704 06:20:31.340168 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.22:44545->172.30.0.10:53: read: connection refused I0704 06:20:31.401435 1 tasks_processing.go:74] worker 16 stopped. E0704 06:20:31.401458 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0704 06:20:31.401465 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0704 06:20:31.401468 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0704 06:20:31.401480 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0704 06:20:31.401521 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=f73fcd48d6f9ebbc83253a1b5f23ef59d0683465ec46e474389103c717467dfb I0704 06:20:31.401529 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0704 06:20:31.401534 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0704 06:20:31.401538 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0704 06:20:31.401575 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0704 06:20:31.401584 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0704 06:20:31.401589 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 462.948429ms to process 7 records I0704 06:20:31.425987 1 tasks_processing.go:74] worker 27 stopped. E0704 06:20:31.426022 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0704 06:20:31.426109 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2rb58ok5g2jcqqlt8f0qks7jvk25h0q7-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2rb58ok5g2jcqqlt8f0qks7jvk25h0q7-primary-cert-bundle-secret" not found I0704 06:20:31.426201 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=7ad9aff8d261612f647ca34320f5376cc853e9ccfa7e05262348905076484dcb I0704 06:20:31.426221 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 486.95156ms to process 1 records I0704 06:20:31.567916 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0704 06:20:31.567946 1 tasks_processing.go:74] worker 29 stopped. I0704 06:20:31.567962 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 595.162921ms to process 0 records I0704 06:20:31.596630 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0704 06:20:31.692896 1 gather_cluster_operator_pods_and_events.go:121] Found 14 pods with 15 containers I0704 06:20:31.692908 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1677721 bytes I0704 06:20:31.693607 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-67z7k pod in namespace openshift-dns (previous: false). I0704 06:20:31.837125 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-67z7k pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-67z7k\" is waiting to start: ContainerCreating" I0704 06:20:31.837143 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-67z7k\" is waiting to start: ContainerCreating" I0704 06:20:31.837151 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-67z7k pod in namespace openshift-dns (previous: false). I0704 06:20:32.012265 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-67z7k pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-67z7k\" is waiting to start: ContainerCreating" I0704 06:20:32.012285 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-67z7k\" is waiting to start: ContainerCreating" I0704 06:20:32.012299 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-f2xfz pod in namespace openshift-dns (previous: false). W0704 06:20:32.020153 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0704 06:20:32.236157 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for node-resolver-f2xfz pod in namespace openshift-dns for failing operator dns-node-resolver (previous: false): "container \"dns-node-resolver\" in pod \"node-resolver-f2xfz\" is waiting to start: ContainerCreating" I0704 06:20:32.236178 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns-node-resolver\" in pod \"node-resolver-f2xfz\" is waiting to start: ContainerCreating" I0704 06:20:32.236190 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-h2wfk pod in namespace openshift-dns (previous: false). I0704 06:20:32.411923 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 06:20:32.411943 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-wgvxb pod in namespace openshift-dns (previous: false). I0704 06:20:32.583862 1 tasks_processing.go:74] worker 31 stopped. I0704 06:20:32.583909 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0704 06:20:32.583924 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 1.560486341s to process 1 records I0704 06:20:32.630614 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 06:20:32.630638 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-657d88cbbb-vdrcg pod in namespace openshift-image-registry (previous: false). I0704 06:20:32.807037 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 06:20:32.807060 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-657d88cbbb-zzsjl pod in namespace openshift-image-registry (previous: false). I0704 06:20:32.898527 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0704 06:20:33.006246 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 06:20:33.006267 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-867579856f-82njd pod in namespace openshift-image-registry (previous: false). W0704 06:20:33.020334 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0704 06:20:33.212066 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-867579856f-82njd pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-867579856f-82njd\" is waiting to start: ContainerCreating" I0704 06:20:33.212084 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-867579856f-82njd\" is waiting to start: ContainerCreating" I0704 06:20:33.212096 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-pbjhd pod in namespace openshift-image-registry (previous: false). I0704 06:20:33.420046 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 06:20:33.420080 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-qlbhx pod in namespace openshift-image-registry (previous: false). I0704 06:20:33.501583 1 tasks_processing.go:74] worker 6 stopped. I0704 06:20:33.501641 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=461670edca4000eaa9bc1af7fb8080156f72cff0173d28e0c8433bf27b2ab90e I0704 06:20:33.501697 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=33f7c9f789e7cea50fb27cdd53ec2cdfa8c5dd9fc35b07a53d56e94da2492a3d I0704 06:20:33.501732 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0704 06:20:33.501760 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=a9bb0ecd1a3d8a7415dfc45db6ef8a2c67f30eea1f4a45f045633200700675e2 I0704 06:20:33.501778 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0704 06:20:33.501820 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=67d72d3c6ed43176f24e9799e83661e46d0c6af62c4b138a949038dc5eac44be I0704 06:20:33.501851 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=a74207ee0129fe2d85454701d1dbb2b2a40601c12f5beb4f1fac3e1012a1c4b3 I0704 06:20:33.501876 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=3d69085b58683318070030e36b2925b59d390117cb658bcd1a9661e8b9a72b7f I0704 06:20:33.501907 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=d9b72885f8bdcea431bc6a4cf7d5c9ba5044ead671787a7e091f121efebd0034 I0704 06:20:33.501917 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=e5ff11d57817f84a678f6fa9565af55bd1120227c16a21933637ab62675a6d70 I0704 06:20:33.501934 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=ac2b8859990684002c86b5059cc6d9ff927188aef6302c3dc76c90ac7aa7d627 I0704 06:20:33.501943 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0704 06:20:33.501958 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=94bd76309e1bfadfe3148dacf24e569f9deb8e5c586ee71b494c0475acac4f24 I0704 06:20:33.501968 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0704 06:20:33.501984 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=b3c22a1974390fe8a571ef789b5dab4e6e7c1e68c66828985b0fa221e38c61e3 I0704 06:20:33.501993 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0704 06:20:33.502011 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=a392fe287d64dcddc75843bc5559f8c92c026edd4de1deabb5f58811a58d8b1c I0704 06:20:33.502020 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0704 06:20:33.502035 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=9ae2c03901e028b958864b503509638e4e9f2f4c969cec0c3c6b26fb986dd8fd I0704 06:20:33.502166 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=0e9ca76c2cefe7e59bc859fe8000b8f96c3ec1e9eb9e1c32657da5db170d8db2 I0704 06:20:33.502176 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0704 06:20:33.502183 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0704 06:20:33.502203 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0704 06:20:33.502224 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=a7efc3162ba2a281d5c592c3f0c4729e0b4aabb8671a93ebf82a3b335c6e5eab I0704 06:20:33.502247 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=f3d2f5701e773232053e49a6fba1c0a09579daff9e6d2b6e2bff862547019d7d I0704 06:20:33.502256 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0704 06:20:33.502272 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=1e50ea0c87f58c6bbde576754c085ead299f069aafaf2446e1630e23801860a4 I0704 06:20:33.502281 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0704 06:20:33.502294 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=4d01dc0700da2f21230ae85a3c6604d6555aa436a9a178405e148717e4b8abe0 I0704 06:20:33.502307 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=ed5557579e41a8002b7f0dd89ee4e26b6b2a718dc9d5078ebefcd5876d54540f I0704 06:20:33.502321 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=588cc5d21fa5b83e5cd0104d580069902c550cd2c6fe1671d1f4461f2b34d7c6 I0704 06:20:33.502336 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=14f3ce9803c3ce8fe44f542929ac913bc3ea486e30c22c918434b63d311e940f I0704 06:20:33.502358 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=c6dd8e4162e47caeb7dcce1bd536b62fa5c3ef7acac4182193c121c8f3a596b6 I0704 06:20:33.502367 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0704 06:20:33.502393 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=3adfe74970a8b60b2d2b425eab06329cb1f744a1dbcf70bdbdff6a37fb424acb I0704 06:20:33.502409 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0704 06:20:33.502417 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0704 06:20:33.502424 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.481309301s to process 37 records I0704 06:20:33.543884 1 tasks_processing.go:74] worker 21 stopped. I0704 06:20:33.543905 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 2.603722357s to process 0 records I0704 06:20:33.613393 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for node-ca-qlbhx pod in namespace openshift-image-registry for failing operator node-ca (previous: false): "container \"node-ca\" in pod \"node-ca-qlbhx\" is waiting to start: ContainerCreating" I0704 06:20:33.613416 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"node-ca\" in pod \"node-ca-qlbhx\" is waiting to start: ContainerCreating" I0704 06:20:33.613429 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-vhcnm pod in namespace openshift-image-registry (previous: false). I0704 06:20:33.814754 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 06:20:33.814775 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5465dd5d55-f498m pod in namespace openshift-ingress (previous: false). I0704 06:20:34.010218 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0704 06:20:34.010238 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5465dd5d55-jr59h pod in namespace openshift-ingress (previous: false). W0704 06:20:34.020309 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0704 06:20:34.211816 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5465dd5d55-jr59h pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5465dd5d55-jr59h\" is waiting to start: ContainerCreating" I0704 06:20:34.211836 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5465dd5d55-jr59h\" is waiting to start: ContainerCreating" I0704 06:20:34.211850 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7864c7867f-pr7t2 pod in namespace openshift-ingress (previous: false). I0704 06:20:34.411896 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-7864c7867f-pr7t2 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-7864c7867f-pr7t2\" is waiting to start: ContainerCreating" I0704 06:20:34.411919 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-7864c7867f-pr7t2\" is waiting to start: ContainerCreating" I0704 06:20:34.411933 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-cd77h pod in namespace openshift-ingress-canary (previous: false). I0704 06:20:34.611562 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-cd77h pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-cd77h\" is waiting to start: ContainerCreating" I0704 06:20:34.611584 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-cd77h\" is waiting to start: ContainerCreating" I0704 06:20:34.611601 1 tasks_processing.go:74] worker 30 stopped. I0704 06:20:34.611731 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=028dcf6bd4dd71b2e420a77900afa772d1016aa586e6619dc33bef8fd351b921 I0704 06:20:34.611779 1 recorder.go:75] Recording events/openshift-dns with fingerprint=fd097859ffbbb3daf463754f31bc4cca651a24f5ef13fafa60b78d5da5d0ca15 I0704 06:20:34.611881 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=4a1fd4371fc087ecc7a401cd45ae2dfa05bcfa38004c019690901a49e0ac44b7 I0704 06:20:34.611919 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=db046ad16ac24346ce6ffea3fee51f115b22d119a50241209dab2bf03d4793fd I0704 06:20:34.611973 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=e67b37e03ec29ba38a675493b08bf26c54bbf06cf21fb2dbe057b7f0de9d5e87 I0704 06:20:34.611991 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=731a789dbee650a599f3c49644b3e16054ef4bb9ba68075bc15d8e2347b98800 I0704 06:20:34.612002 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 3.58932226s to process 6 records W0704 06:20:35.020347 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0704 06:20:36.021458 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0704 06:20:36.021493 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0704 06:20:36.021514 1 tasks_processing.go:74] worker 11 stopped. E0704 06:20:36.021527 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0704 06:20:36.021541 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0704 06:20:36.021561 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0704 06:20:36.021574 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.08118133s to process 1 records I0704 06:20:43.663394 1 tasks_processing.go:74] worker 23 stopped. I0704 06:20:43.663443 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0704 06:20:43.663459 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.722614161s to process 1 records I0704 06:20:44.159728 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 06:20:44.618951 1 tasks_processing.go:74] worker 4 stopped. I0704 06:20:44.619260 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=9d475996b970b5499163b99cb7866431c26bc05ed77261e8b402c7515458063c I0704 06:20:44.619284 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.606546791s to process 1 records E0704 06:20:44.619366 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.681s with: function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0704 06:20:44.620541 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0704 06:20:44.620557 1 periodic.go:209] Running workloads gatherer I0704 06:20:44.620587 1 tasks_processing.go:45] number of workers: 2 I0704 06:20:44.620595 1 tasks_processing.go:69] worker 1 listening for tasks. I0704 06:20:44.620601 1 tasks_processing.go:71] worker 1 working on workload_info task. I0704 06:20:44.620608 1 tasks_processing.go:69] worker 0 listening for tasks. I0704 06:20:44.620618 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0704 06:20:44.646943 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 06:20:44.648215 1 tasks_processing.go:74] worker 0 stopped. I0704 06:20:44.648296 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 27.583134ms to process 0 records I0704 06:20:44.741077 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 21s for image data I0704 06:20:44.751613 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (12ms) I0704 06:20:44.760741 1 gather_workloads_info.go:387] No image sha256:80748ba08e1c264a8c105e7f607eff386a66378e024443a844993ee9292858c1 (9ms) I0704 06:20:44.769767 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (9ms) I0704 06:20:44.778589 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (9ms) I0704 06:20:44.787076 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (8ms) I0704 06:20:44.795319 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (8ms) I0704 06:20:44.803462 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (8ms) I0704 06:20:44.811602 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (8ms) I0704 06:20:44.820035 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (8ms) I0704 06:20:44.828169 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (8ms) I0704 06:20:44.846777 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 06:20:44.852150 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (24ms) I0704 06:20:44.950449 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (98ms) I0704 06:20:45.050870 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (100ms) I0704 06:20:45.149689 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (99ms) I0704 06:20:45.251046 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (101ms) I0704 06:20:45.350435 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (99ms) I0704 06:20:45.451056 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (101ms) I0704 06:20:45.554151 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (103ms) I0704 06:20:45.655482 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (101ms) I0704 06:20:45.752872 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (97ms) I0704 06:20:45.850526 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (98ms) I0704 06:20:45.949625 1 gather_workloads_info.go:387] No image sha256:95418d0ba6fc0aa96647d97cf6530967f7d7ae97a82ce7f7e41c8e44310b0b72 (99ms) I0704 06:20:46.054928 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (105ms) I0704 06:20:46.156720 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (102ms) I0704 06:20:46.250441 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (94ms) I0704 06:20:46.350323 1 gather_workloads_info.go:387] No image sha256:fac526ab9335a9a8decb028a30673f4ae718deae8da114f36103b6438ba6458f (100ms) I0704 06:20:46.450069 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0704 06:20:46.550110 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0704 06:20:46.650224 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0704 06:20:46.750726 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (100ms) I0704 06:20:46.850872 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (100ms) I0704 06:20:46.951033 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0704 06:20:47.049901 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (99ms) I0704 06:20:47.049935 1 tasks_processing.go:74] worker 1 stopped. E0704 06:20:47.049945 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0704 06:20:47.050323 1 recorder.go:75] Recording config/workload_info with fingerprint=5dd86caea1201d38f6f652b407ffb803e74930754798240e4680532a3ab4340e I0704 06:20:47.050345 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.429326313s to process 1 records E0704 06:20:47.050377 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.429s with: function \"workload_info\" failed with an error" I0704 06:20:47.051492 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0704 06:20:47.051508 1 periodic.go:209] Running conditional gatherer I0704 06:20:47.057783 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0704 06:20:47.065786 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.22:48852->172.30.0.10:53: read: connection refused E0704 06:20:47.066095 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0704 06:20:47.066167 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0704 06:20:47.072765 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0704 06:20:47.072782 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 06:20:47.072788 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 06:20:47.072793 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 06:20:47.072798 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 06:20:47.072802 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 06:20:47.072806 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 06:20:47.072810 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 06:20:47.072815 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 06:20:47.072821 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0704 06:20:47.072852 1 tasks_processing.go:45] number of workers: 3 I0704 06:20:47.072878 1 tasks_processing.go:69] worker 2 listening for tasks. I0704 06:20:47.072887 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0704 06:20:47.072887 1 tasks_processing.go:69] worker 0 listening for tasks. I0704 06:20:47.072900 1 tasks_processing.go:69] worker 1 listening for tasks. I0704 06:20:47.072907 1 tasks_processing.go:74] worker 1 stopped. I0704 06:20:47.072920 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0704 06:20:47.072925 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0704 06:20:47.073012 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0704 06:20:47.073032 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.404µs to process 1 records I0704 06:20:47.073074 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0704 06:20:47.073084 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.146µs to process 1 records I0704 06:20:47.073093 1 tasks_processing.go:74] worker 0 stopped. I0704 06:20:47.073313 1 tasks_processing.go:74] worker 2 stopped. I0704 06:20:47.073329 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 366.538µs to process 0 records I0704 06:20:47.073357 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.22:48852->172.30.0.10:53: read: connection refused I0704 06:20:47.073385 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0704 06:20:47.095907 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=2333d09b4a129cf88dc91cab25bfb22b02066b51df8c3cf8ab28b4d04aec1655 I0704 06:20:47.096050 1 diskrecorder.go:70] Writing 102 records to /var/lib/insights-operator/insights-2026-07-04-062047.tar.gz I0704 06:20:47.103022 1 diskrecorder.go:51] Wrote 102 records to disk in 7ms I0704 06:20:47.103051 1 periodic.go:278] Gathering cluster info every 2h0m0s I0704 06:20:47.103070 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0704 06:20:57.305776 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 06:22:00.219238 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="afc0681d7f1f05127e56c3d0f2ffc5b29c77835a3b3ad774a1322553690d10bd") W0704 06:22:00.219269 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0704 06:22:00.219319 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="4d5e2eb96dfaa44525ba1080451dfbeac7cbcd566573d5c067d22c6b29413f05") I0704 06:22:00.219337 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0704 06:22:00.219368 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0704 06:22:00.219389 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0704 06:22:00.219416 1 base_controller.go:181] Shutting down LoggingSyncer ... I0704 06:22:00.219431 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="f7933e3be6aa18cab9238e3c2249e519869813cc40a312b23dc3ba1ac9cfae35") I0704 06:22:00.219448 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0704 06:22:00.219452 1 base_controller.go:181] Shutting down ConfigController ... I0704 06:22:00.219456 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0704 06:22:00.219469 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0704 06:22:00.219478 1 base_controller.go:113] All ConfigController workers have been terminated