I0704 22:25:31.447163 1 cmd.go:253] Using service-serving-cert provided certificates I0704 22:25:31.447673 1 observer_polling.go:159] Starting file observer I0704 22:25:31.873548 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0704 22:25:31.873760 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0704 22:25:31.874825 1 secure_serving.go:57] Forcing use of http/1.1 only W0704 22:25:31.874923 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0704 22:25:31.874959 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0704 22:25:31.874984 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0704 22:25:31.875006 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0704 22:25:31.875036 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0704 22:25:31.875058 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0704 22:25:31.874862 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0704 22:25:31.879985 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0704 22:25:31.880002 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"247b462b-1264-4c4c-9a1d-9b2d9b9d9a12", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0704 22:25:31.882350 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0704 22:25:31.882365 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0704 22:25:31.882375 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0704 22:25:31.882373 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0704 22:25:31.882380 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0704 22:25:31.882385 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0704 22:25:31.882675 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0704 22:25:31.882956 1 secure_serving.go:213] Serving securely on [::]:8443 I0704 22:25:31.882981 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0704 22:25:31.887035 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0704 22:25:31.887057 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0704 22:25:31.887164 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0704 22:25:31.894277 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0704 22:25:31.894294 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0704 22:25:31.900713 1 secretconfigobserver.go:119] support secret does not exist I0704 22:25:31.907367 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0704 22:25:31.913721 1 secretconfigobserver.go:119] support secret does not exist I0704 22:25:31.916731 1 recorder.go:161] Pruning old reports every 8h8m15s, max age is 288h0m0s I0704 22:25:31.923551 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0704 22:25:31.923569 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0704 22:25:31.923573 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0704 22:25:31.923581 1 insightsreport.go:296] Starting report retriever I0704 22:25:31.923589 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0704 22:25:31.923606 1 periodic.go:209] Running clusterconfig gatherer I0704 22:25:31.923660 1 tasks_processing.go:45] number of workers: 32 I0704 22:25:31.923685 1 tasks_processing.go:69] worker 0 listening for tasks. I0704 22:25:31.923693 1 tasks_processing.go:69] worker 31 listening for tasks. I0704 22:25:31.923695 1 tasks_processing.go:69] worker 9 listening for tasks. I0704 22:25:31.923700 1 tasks_processing.go:71] worker 31 working on nodes task. I0704 22:25:31.923705 1 tasks_processing.go:69] worker 1 listening for tasks. I0704 22:25:31.923708 1 tasks_processing.go:69] worker 20 listening for tasks. I0704 22:25:31.923714 1 tasks_processing.go:69] worker 6 listening for tasks. I0704 22:25:31.923710 1 tasks_processing.go:69] worker 21 listening for tasks. I0704 22:25:31.923720 1 tasks_processing.go:69] worker 10 listening for tasks. I0704 22:25:31.923717 1 tasks_processing.go:69] worker 22 listening for tasks. I0704 22:25:31.923724 1 tasks_processing.go:69] worker 23 listening for tasks. I0704 22:25:31.923728 1 tasks_processing.go:69] worker 7 listening for tasks. I0704 22:25:31.923730 1 tasks_processing.go:69] worker 24 listening for tasks. I0704 22:25:31.923735 1 tasks_processing.go:69] worker 8 listening for tasks. I0704 22:25:31.923736 1 tasks_processing.go:69] worker 5 listening for tasks. I0704 22:25:31.923744 1 tasks_processing.go:69] worker 27 listening for tasks. I0704 22:25:31.923739 1 tasks_processing.go:69] worker 16 listening for tasks. I0704 22:25:31.923751 1 tasks_processing.go:69] worker 28 listening for tasks. I0704 22:25:31.923754 1 tasks_processing.go:69] worker 26 listening for tasks. I0704 22:25:31.923743 1 tasks_processing.go:69] worker 29 listening for tasks. I0704 22:25:31.923759 1 tasks_processing.go:69] worker 18 listening for tasks. I0704 22:25:31.923762 1 tasks_processing.go:69] worker 2 listening for tasks. I0704 22:25:31.923763 1 tasks_processing.go:69] worker 19 listening for tasks. I0704 22:25:31.923769 1 tasks_processing.go:69] worker 13 listening for tasks. I0704 22:25:31.923693 1 tasks_processing.go:71] worker 0 working on openstack_controlplanes task. I0704 22:25:31.923774 1 tasks_processing.go:69] worker 14 listening for tasks. I0704 22:25:31.923773 1 tasks_processing.go:69] worker 3 listening for tasks. I0704 22:25:31.923783 1 tasks_processing.go:69] worker 30 listening for tasks. I0704 22:25:31.923738 1 tasks_processing.go:69] worker 15 listening for tasks. I0704 22:25:31.923787 1 tasks_processing.go:69] worker 4 listening for tasks. I0704 22:25:31.923749 1 tasks_processing.go:69] worker 11 listening for tasks. I0704 22:25:31.923756 1 tasks_processing.go:69] worker 12 listening for tasks. I0704 22:25:31.923795 1 tasks_processing.go:71] worker 4 working on proxies task. I0704 22:25:31.923798 1 tasks_processing.go:71] worker 1 working on infrastructures task. I0704 22:25:31.923800 1 tasks_processing.go:71] worker 22 working on install_plans task. I0704 22:25:31.923803 1 tasks_processing.go:71] worker 6 working on schedulers task. I0704 22:25:31.923747 1 tasks_processing.go:69] worker 17 listening for tasks. I0704 22:25:31.923794 1 tasks_processing.go:71] worker 9 working on machines task. I0704 22:25:31.923886 1 tasks_processing.go:71] worker 21 working on config_maps task. I0704 22:25:31.924000 1 tasks_processing.go:71] worker 17 working on support_secret task. I0704 22:25:31.924037 1 tasks_processing.go:71] worker 28 working on storage_classes task. I0704 22:25:31.924048 1 tasks_processing.go:71] worker 18 working on aggregated_monitoring_cr_names task. I0704 22:25:31.924039 1 tasks_processing.go:71] worker 19 working on dvo_metrics task. I0704 22:25:31.923701 1 tasks_processing.go:69] worker 25 listening for tasks. I0704 22:25:31.924186 1 tasks_processing.go:71] worker 25 working on machine_sets task. I0704 22:25:31.924262 1 tasks_processing.go:71] worker 2 working on openstack_version task. I0704 22:25:31.924311 1 tasks_processing.go:71] worker 13 working on ingress_certificates task. I0704 22:25:31.924347 1 tasks_processing.go:71] worker 14 working on mutating_webhook_configurations task. I0704 22:25:31.924466 1 tasks_processing.go:71] worker 12 working on machine_configs task. I0704 22:25:31.923795 1 tasks_processing.go:71] worker 15 working on openshift_machine_api_events task. I0704 22:25:31.924573 1 tasks_processing.go:71] worker 29 working on sap_config task. I0704 22:25:31.924669 1 tasks_processing.go:71] worker 27 working on certificate_signing_requests task. I0704 22:25:31.923790 1 tasks_processing.go:71] worker 10 working on container_images task. I0704 22:25:31.924341 1 tasks_processing.go:71] worker 3 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0704 22:25:31.924031 1 tasks_processing.go:71] worker 11 working on pod_network_connectivity_checks task. I0704 22:25:31.924044 1 tasks_processing.go:71] worker 26 working on lokistack task. I0704 22:25:31.924560 1 tasks_processing.go:71] worker 23 working on nodenetworkstates task. I0704 22:25:31.925060 1 tasks_processing.go:71] worker 5 working on cluster_apiserver task. I0704 22:25:31.925103 1 tasks_processing.go:71] worker 16 working on openshift_logging task. I0704 22:25:31.924567 1 tasks_processing.go:71] worker 24 working on machine_autoscalers task. I0704 22:25:31.924546 1 tasks_processing.go:71] worker 30 working on image_registries task. I0704 22:25:31.924564 1 tasks_processing.go:71] worker 7 working on qemu_kubevirt_launcher_logs task. I0704 22:25:31.924557 1 tasks_processing.go:71] worker 8 working on machine_healthchecks task. I0704 22:25:31.923848 1 tasks_processing.go:71] worker 20 working on service_accounts task. I0704 22:25:31.928121 1 tasks_processing.go:71] worker 9 working on machine_config_pools task. E0704 22:25:31.928130 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0704 22:25:31.928142 1 gather.go:177] gatherer "clusterconfig" function "machines" took 4.243086ms to process 0 records I0704 22:25:31.929339 1 tasks_processing.go:71] worker 0 working on crds task. I0704 22:25:31.929345 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 5.535281ms to process 0 records I0704 22:25:31.930817 1 tasks_processing.go:71] worker 25 working on ceph_cluster task. I0704 22:25:31.930841 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 6.600242ms to process 0 records I0704 22:25:31.931292 1 tasks_processing.go:71] worker 29 working on image_pruners task. I0704 22:25:31.931300 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 6.698375ms to process 0 records I0704 22:25:31.931502 1 tasks_processing.go:71] worker 11 working on cost_management_metrics_configs task. E0704 22:25:31.931512 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0704 22:25:31.931541 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 6.630996ms to process 0 records I0704 22:25:31.931557 1 tasks_processing.go:71] worker 23 working on networks task. I0704 22:25:31.931563 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 6.532532ms to process 0 records I0704 22:25:31.931570 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 7.247693ms to process 0 records I0704 22:25:31.931577 1 tasks_processing.go:71] worker 2 working on operators task. I0704 22:25:31.932125 1 tasks_processing.go:71] worker 16 working on authentication task. I0704 22:25:31.932139 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 7.000385ms to process 0 records I0704 22:25:31.932401 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0704 22:25:31.932416 1 controller.go:326] The initial operator extension status is healthy I0704 22:25:31.932424 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0704 22:25:31.932427 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0704 22:25:31.932430 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0704 22:25:31.932443 1 controller.go:489] The operator is still being initialized I0704 22:25:31.932450 1 controller.go:512] The operator is healthy I0704 22:25:31.932547 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0704 22:25:31.932551 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0704 22:25:31.932653 1 operator.go:288] started I0704 22:25:31.932683 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0704 22:25:31.932770 1 tasks_processing.go:71] worker 26 working on tsdb_status task. I0704 22:25:31.932780 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 7.753545ms to process 0 records I0704 22:25:31.933179 1 tasks_processing.go:71] worker 1 working on clusterroles task. I0704 22:25:31.933826 1 recorder.go:75] Recording config/infrastructure with fingerprint=513f740635877920ce68d38314f697a53b44ea67db20df66a8d482f6bd082488 I0704 22:25:31.933845 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 9.370725ms to process 1 records I0704 22:25:31.933962 1 tasks_processing.go:71] worker 4 working on metrics task. I0704 22:25:31.934052 1 recorder.go:75] Recording config/proxy with fingerprint=3120283e9d02b06c6a01fdaa13161e4e8f443756e4257ad88a9f55608046337c I0704 22:25:31.934085 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 10.152136ms to process 1 records I0704 22:25:31.934102 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 8.885597ms to process 0 records I0704 22:25:31.934120 1 tasks_processing.go:71] worker 24 working on openstack_dataplanedeployments task. I0704 22:25:31.934394 1 tasks_processing.go:71] worker 8 working on version task. E0704 22:25:31.934554 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0704 22:25:31.934573 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 9.103831ms to process 0 records I0704 22:25:31.935492 1 tasks_processing.go:71] worker 25 working on silenced_alerts task. I0704 22:25:31.935639 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 4.6602ms to process 0 records I0704 22:25:31.935913 1 tasks_processing.go:71] worker 11 working on monitoring_persistent_volumes task. I0704 22:25:31.936099 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 4.396505ms to process 0 records I0704 22:25:31.938207 1 tasks_processing.go:71] worker 9 working on oauths task. I0704 22:25:31.938218 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 10.07197ms to process 0 records E0704 22:25:31.938254 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.94.209:9091: connect: connection refused I0704 22:25:31.938273 1 tasks_processing.go:71] worker 26 working on sap_datahubs task. E0704 22:25:31.938293 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.94.209:9091: connect: connection refused E0704 22:25:31.938302 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.94.209:9091: connect: connection refused I0704 22:25:31.938316 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 5.491052ms to process 0 records E0704 22:25:31.938335 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.94.209:9091: connect: connection refused I0704 22:25:31.938358 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 4.330122ms to process 0 records I0704 22:25:31.938377 1 tasks_processing.go:71] worker 4 working on nodenetworkconfigurationpolicies task. I0704 22:25:31.938461 1 tasks_processing.go:71] worker 15 working on ingress task. I0704 22:25:31.938494 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 13.884221ms to process 0 records I0704 22:25:31.938911 1 tasks_processing.go:71] worker 6 working on validating_webhook_configurations task. I0704 22:25:31.939004 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=6500714573c78e906cce3edf8b49e82cca2d45b0c02d61a9550fe269b62e2dfa I0704 22:25:31.939021 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 15.092384ms to process 1 records I0704 22:25:31.940421 1 tasks_processing.go:71] worker 17 working on jaegers task. E0704 22:25:31.940431 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0704 22:25:31.940445 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 16.374827ms to process 0 records I0704 22:25:31.940855 1 tasks_processing.go:71] worker 5 working on sap_pods task. I0704 22:25:31.941064 1 recorder.go:75] Recording config/apiserver with fingerprint=fe44656f0fb3e5fab700903ac8f0fe64510ccab95dcbc30f7dab87128b9846d1 I0704 22:25:31.941081 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 15.753813ms to process 1 records I0704 22:25:31.941934 1 tasks_processing.go:71] worker 31 working on olm_operators task. I0704 22:25:31.942202 1 recorder.go:75] Recording config/node/ip-10-0-0-20.ec2.internal with fingerprint=7fb317d257068fb422808acd7418aec68415d9d0652eb250dd73483e8e604b15 I0704 22:25:31.942268 1 recorder.go:75] Recording config/node/ip-10-0-1-215.ec2.internal with fingerprint=cc62612d75fda9d0e90ec1648e9d84ed42c34138223aa7adad9abd6187b2d25f I0704 22:25:31.942319 1 recorder.go:75] Recording config/node/ip-10-0-2-53.ec2.internal with fingerprint=35791be30e31767a1085a3231d109537c48eb14aa543cda00fec16ce9de51bc3 I0704 22:25:31.942327 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 18.220562ms to process 3 records I0704 22:25:31.946305 1 tasks_processing.go:71] worker 29 working on overlapping_namespace_uids task. I0704 22:25:31.946618 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=0ff4c0da6cfff6490b354bbe1385335f316985d397d19f34de176b2427bec3a0 I0704 22:25:31.946634 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 15.000186ms to process 1 records I0704 22:25:31.946729 1 tasks_processing.go:71] worker 23 working on node_logs task. I0704 22:25:31.946766 1 recorder.go:75] Recording config/network with fingerprint=ea0b8e9c17db9f375eac70d430f41ff5f250db999fb70009a342146578e90666 I0704 22:25:31.946780 1 gather.go:177] gatherer "clusterconfig" function "networks" took 14.79659ms to process 1 records I0704 22:25:31.946865 1 tasks_processing.go:71] worker 16 working on feature_gates task. I0704 22:25:31.946967 1 recorder.go:75] Recording config/authentication with fingerprint=50781b8af49138e2b509bc9ab84867a70174a04fc186899bbdd5757042dffddc I0704 22:25:31.947027 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 14.228594ms to process 1 records I0704 22:25:31.947085 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0704 22:25:31.947102 1 controller.go:212] Source scaController *sca.Controller is not ready I0704 22:25:31.947107 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0704 22:25:31.947111 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0704 22:25:31.947116 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0704 22:25:31.947141 1 controller.go:489] The operator is still being initialized I0704 22:25:31.947168 1 tasks_processing.go:71] worker 30 working on storage_cluster task. I0704 22:25:31.947172 1 controller.go:512] The operator is healthy I0704 22:25:31.947551 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=1a0f87355066aa6521572c9f3ad5d1a96a67c94dbd49452cf772ef11e1036d1a I0704 22:25:31.947566 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 21.887816ms to process 1 records I0704 22:25:31.947696 1 tasks_processing.go:71] worker 17 working on operators_pods_and_events task. I0704 22:25:31.947701 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 7.258827ms to process 0 records I0704 22:25:31.947923 1 tasks_processing.go:71] worker 5 working on pdbs task. I0704 22:25:31.947930 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 7.050124ms to process 0 records I0704 22:25:31.952632 1 tasks_processing.go:71] worker 24 working on container_runtime_configs task. I0704 22:25:31.952641 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 18.498379ms to process 0 records I0704 22:25:31.952740 1 tasks_processing.go:71] worker 26 working on image task. I0704 22:25:31.952771 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 14.459342ms to process 0 records W0704 22:25:31.953508 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0704 22:25:31.954107 1 tasks_processing.go:71] worker 4 working on openstack_dataplanenodesets task. I0704 22:25:31.954169 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 15.720451ms to process 0 records I0704 22:25:31.955773 1 tasks_processing.go:71] worker 14 working on active_alerts task. I0704 22:25:31.959123 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=e5c571f4df3d9f6dabeeb88d8b9d97094df0e58b2532458fcacd5f7632ad1437 I0704 22:25:31.959489 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=ad9610c300697c8d1a544bb61c0ad26c5d0b805c6ba55b266851ee8eb9a82a11 I0704 22:25:31.959649 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=44f76d0b6b6c7d7289536070b47e4984a1e6729c5e3fbe3449a00b162119afd0 I0704 22:25:31.959717 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 31.370027ms to process 3 records I0704 22:25:31.959875 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=401e56f110f16d59c8a7ef2883742d1b2afe056ab5ed448c27fea0d02baf7c49 I0704 22:25:31.959944 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=6abee842ee2591493e35d760e70bc027ebbe359f1f13bee9915fe13d858f376a I0704 22:25:31.959987 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 35.062612ms to process 2 records I0704 22:25:31.960021 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 35.128302ms to process 0 records I0704 22:25:31.960053 1 tasks_processing.go:74] worker 27 stopped. I0704 22:25:31.960080 1 tasks_processing.go:74] worker 28 stopped. I0704 22:25:31.961352 1 tasks_processing.go:74] worker 0 stopped. I0704 22:25:31.962228 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=573789d8bfe4d624f0f5cc160a59976bae7ed16bc37aac035bb719f1b728dbc7 I0704 22:25:31.962597 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=78413deef0300257c8fdecfbf2c0edc451cd7fc64976e0ee383e793031665d40 I0704 22:25:31.962617 1 gather.go:177] gatherer "clusterconfig" function "crds" took 31.998041ms to process 2 records I0704 22:25:31.962656 1 gather_logs.go:145] no pods in namespace were found I0704 22:25:31.962666 1 tasks_processing.go:74] worker 7 stopped. I0704 22:25:31.962673 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 37.383625ms to process 0 records I0704 22:25:31.962863 1 tasks_processing.go:74] worker 9 stopped. I0704 22:25:31.963173 1 recorder.go:75] Recording config/oauth with fingerprint=b39f5b159d003fd2d5012001177ca14447092ab7950e4aff8c48da83217e2b8d I0704 22:25:31.963190 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 24.640944ms to process 1 records I0704 22:25:31.963352 1 recorder.go:75] Recording config/ingress with fingerprint=dc448af3313f657af042ef7840f7b7f3947ffdd214147400ee4db5c4b763b6d8 I0704 22:25:31.963409 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 24.468562ms to process 1 records I0704 22:25:31.963424 1 tasks_processing.go:74] worker 15 stopped. I0704 22:25:31.963497 1 tasks_processing.go:74] worker 30 stopped. I0704 22:25:31.963510 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 16.321676ms to process 0 records I0704 22:25:31.963811 1 tasks_processing.go:74] worker 31 stopped. I0704 22:25:31.963855 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 21.863823ms to process 0 records I0704 22:25:31.964669 1 tasks_processing.go:74] worker 24 stopped. I0704 22:25:31.964709 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 12.016541ms to process 0 records I0704 22:25:31.967040 1 tasks_processing.go:74] worker 6 stopped. I0704 22:25:31.967152 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=12945ccd8f79a00717c89041bb54754ddb4e605b7216ef201516eced5f46071f I0704 22:25:31.967250 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=e905531a0d7348418a4916a77ebb1506c9f9390d6f7a67c69e523c8d75b6d5e9 I0704 22:25:31.967376 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=47dcddf6daa0d748b8c210f8379f349f5f7c527cd9685e018027bd6e842571c5 I0704 22:25:31.967418 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=3cbdfb88c45342776c566b619e8d66c3252e07f6e36526872b836fcb009b7791 I0704 22:25:31.967449 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=7c498ddda7693144c2f7f0eefbf8469896cefd09f05c58770c8f57e689288188 I0704 22:25:31.967555 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=ea2d0a64ffd4eba4c0df760e49a23bbc6bd8ebadda73ea70e58dc284a1d1af0e I0704 22:25:31.967607 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=ff2769b9daf4de0f4c23bdbc904d23957aba167f427a3657926d4ab08fac1bf0 I0704 22:25:31.967653 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=d21d81e26d120190a094ad9501c3300f78af051637c6398173587bc66083468d I0704 22:25:31.967700 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=fccc40c01c71f35e736958012d96103d292354646d224fcc1207583bc54d56b2 I0704 22:25:31.967757 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=b4b9451925c0beae9a6ab139aea33fe3a0b5ccb6bed711fecadb16019e0ffb43 I0704 22:25:31.967797 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=884ffab9ee1bbc64b0d39cb15208341419ef5e47a648739937d94d3f768715c1 I0704 22:25:31.967828 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=a5e3848cf44529b87738db14ce876167fb535c3ec7628275b68fbf47232fbeb4 I0704 22:25:31.967872 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=a382ae60d0cb521b174e729cb85a4b31770ad3a920d63ef7f55de40bb7d5d0fa I0704 22:25:31.967881 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 28.10769ms to process 13 records I0704 22:25:31.969787 1 tasks_processing.go:74] worker 29 stopped. I0704 22:25:31.969828 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0704 22:25:31.969848 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 23.424486ms to process 1 records I0704 22:25:31.970439 1 tasks_processing.go:74] worker 1 stopped. I0704 22:25:31.970826 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=81a49e028f7b5633348aa919cfdc2a3807b8c1b14b3eccf09d8b712d732d4e91 I0704 22:25:31.970997 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=ee7d88a7f7020526113616e15cedcbf101e7187ff07808efff86cfe43e81540a I0704 22:25:31.971119 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 37.243129ms to process 2 records I0704 22:25:31.971468 1 tasks_processing.go:74] worker 11 stopped. I0704 22:25:31.971479 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 35.490708ms to process 0 records I0704 22:25:31.971669 1 tasks_processing.go:74] worker 23 stopped. I0704 22:25:31.971692 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 24.921288ms to process 0 records I0704 22:25:31.972042 1 tasks_processing.go:74] worker 12 stopped. I0704 22:25:31.972075 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0704 22:25:31.972090 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 47.552372ms to process 1 records I0704 22:25:31.972101 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 17.920548ms to process 0 records I0704 22:25:31.972113 1 tasks_processing.go:74] worker 4 stopped. I0704 22:25:31.973297 1 tasks_processing.go:74] worker 16 stopped. I0704 22:25:31.973404 1 recorder.go:75] Recording config/featuregate with fingerprint=0ac3ea1b5140284d554b1f442710ff32958f21f4fc4df6d0285e45e48fe21dc9 I0704 22:25:31.973417 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 26.413087ms to process 1 records I0704 22:25:31.977054 1 tasks_processing.go:74] worker 26 stopped. I0704 22:25:31.977149 1 recorder.go:75] Recording config/image with fingerprint=2547228cb9828599020fb5f7a6f77fa50622b9177a2c10f8d25367ec4bd87a09 I0704 22:25:31.977165 1 gather.go:177] gatherer "clusterconfig" function "image" took 24.296505ms to process 1 records I0704 22:25:31.982546 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0704 22:25:31.982553 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0704 22:25:31.982610 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0704 22:25:31.987499 1 base_controller.go:82] Caches are synced for ConfigController I0704 22:25:31.987511 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0704 22:25:31.994626 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0704 22:25:31.996441 1 tasks_processing.go:74] worker 8 stopped. I0704 22:25:31.997163 1 recorder.go:75] Recording config/version with fingerprint=1aaf30c1f89975471a6db1a9e7b02ae56945e1c06f647575616954b5444096a5 I0704 22:25:31.997192 1 recorder.go:75] Recording config/id with fingerprint=82a3162a6d398a1b6e19f9c27ec1bd9b97cce30336fbe4a10c2a35685c56b87f I0704 22:25:31.997203 1 gather.go:177] gatherer "clusterconfig" function "version" took 61.992145ms to process 2 records I0704 22:25:31.997214 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0704 22:25:31.997224 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 61.074687ms to process 1 records I0704 22:25:31.997276 1 recorder.go:75] Recording config/alerts with fingerprint=c3c733dfd90b670c72c5f5c721750b17068e843245312b25d361d9a26d8e9ec0 I0704 22:25:31.997327 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 41.264471ms to process 1 records I0704 22:25:31.997364 1 tasks_processing.go:74] worker 14 stopped. I0704 22:25:31.997389 1 tasks_processing.go:74] worker 25 stopped. I0704 22:25:32.006410 1 tasks_processing.go:74] worker 5 stopped. I0704 22:25:32.006494 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=f56367df5ae5af18d0cef26d327650ce3ca44eb683cb0721ee0f2d9179fecc51 I0704 22:25:32.006512 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=bd90b7054ad1d398468e66736259f5cde858a8bbb7a18b24880a794ac0ba2406 I0704 22:25:32.006546 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=a9e83b96f53d1a60016407f75fb7f9c25631d0059ee26c4471723a2b2f3bec08 I0704 22:25:32.006562 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=9d6cbdea06209325ceb154ed05c331eadb74dbfe5ca85ef46c0c7b1d8e6b06d7 I0704 22:25:32.006577 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=5969408291e8b68db6ae56e44c399438bddb91fbe5343420deae5a1c7daac169 I0704 22:25:32.006591 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=92c6502ee6d05c962ad2051c795c311930837f4c8830035e0b1e3790b07b8999 I0704 22:25:32.006604 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=b94dd72b8822f3b664853b322a5072cb4ce899afd92c5c15919f2600461b80e7 I0704 22:25:32.006617 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=d68c3b284e8dffcfc741cae4b986919d717fd3b77c05b653a5ecac23c3d22236 I0704 22:25:32.006631 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=631837fd3d4a62f032ca4c78cf5a7fe0abde9e92e0f92816b63da7bb7ef00a78 I0704 22:25:32.006645 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=79ab42ea49a027448add1830456d8f918da0888360e124135b52ac019750450f I0704 22:25:32.006659 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=3c83347f2e3d468f200837269fc5aaf3982d360c1de5209353214ff3204d023a I0704 22:25:32.006667 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 58.469492ms to process 11 records I0704 22:25:32.006784 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 22:25:32.007103 1 tasks_processing.go:74] worker 18 stopped. I0704 22:25:32.007117 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 83.040687ms to process 0 records I0704 22:25:32.019055 1 tasks_processing.go:74] worker 10 stopped. I0704 22:25:32.020311 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-j77mc with fingerprint=648ca9d6b896f61edf36e080a896441013f8b6514db3c451e435ec0a0fb6432a I0704 22:25:32.020395 1 recorder.go:75] Recording config/pod/openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-2-53.ec2.internal with fingerprint=f01da6ede0ade7b93314fd37f81245a43500afd00a9c7bc1be7a12215ff7ac80 I0704 22:25:32.020464 1 recorder.go:75] Recording config/running_containers with fingerprint=2075bb8e7be716ea64dfaa1cae672a28728b365865f049c7adb63665200f3898 I0704 22:25:32.020487 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 94.212302ms to process 3 records I0704 22:25:32.021826 1 tasks_processing.go:74] worker 3 stopped. I0704 22:25:32.021842 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 96.962696ms to process 0 records I0704 22:25:32.027447 1 tasks_processing.go:74] worker 21 stopped. E0704 22:25:32.027462 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0704 22:25:32.027474 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0704 22:25:32.027479 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0704 22:25:32.027490 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0704 22:25:32.027548 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=f73fcd48d6f9ebbc83253a1b5f23ef59d0683465ec46e474389103c717467dfb I0704 22:25:32.027558 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0704 22:25:32.027563 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0704 22:25:32.027579 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0704 22:25:32.027618 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0704 22:25:32.027626 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0704 22:25:32.027631 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 103.52631ms to process 7 records I0704 22:25:32.032652 1 tasks_processing.go:74] worker 13 stopped. E0704 22:25:32.032666 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0704 22:25:32.032672 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2rbj8kkd6dnnd83bonsq9egj3c78vhkp-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2rbj8kkd6dnnd83bonsq9egj3c78vhkp-primary-cert-bundle-secret" not found I0704 22:25:32.032720 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=07023fa203c6b494e1a0c0accb223c9519b950bda1db3ef5180df407f801718e I0704 22:25:32.032732 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 108.297363ms to process 1 records I0704 22:25:32.032751 1 base_controller.go:82] Caches are synced for LoggingSyncer I0704 22:25:32.032758 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0704 22:25:32.064540 1 cluster_transfer.go:102] no available accepted cluster transfer I0704 22:25:32.064555 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0704 22:25:32.075034 1 gather_cluster_operator_pods_and_events.go:121] Found 6 pods with 6 containers I0704 22:25:32.075045 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 4194304 bytes I0704 22:25:32.075591 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-79b6b98f5c-xlh24 pod in namespace openshift-ingress (previous: false). I0704 22:25:32.109185 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0704 22:25:32.117307 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0704 22:25:32.119611 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7fcb656854-6x5tq pod in namespace openshift-ingress (previous: false). I0704 22:25:32.180692 1 sca.go:266] etc-pki-entitlement secret successfully created I0704 22:25:32.180713 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0704 22:25:32.197016 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7fcb656854-h4mfx pod in namespace openshift-ingress (previous: false). I0704 22:25:32.213330 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-5rs7n pod in namespace openshift-ingress-canary (previous: false). I0704 22:25:32.247064 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-qnm4r pod in namespace openshift-ingress-canary (previous: false). I0704 22:25:32.262223 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-rztzs pod in namespace openshift-ingress-canary (previous: false). I0704 22:25:32.397200 1 tasks_processing.go:74] worker 17 stopped. I0704 22:25:32.397324 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=14e4fc9568043d8800223b7cc9b2ef706b7e8c332d7a980794045f9777f713e0 I0704 22:25:32.397468 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=50b7c30ff33f6c6e5c97c891a5b8f55affb574a1eb6929ee8f0a7cb3d4f1ea2c I0704 22:25:32.397546 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=40c33e3ad3e16134afcc66555fa87b51bbf9abc6099aa680fe22bb056359234e I0704 22:25:32.397599 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-79b6b98f5c-xlh24/router_current.log with fingerprint=a142e0c073e300e8e4a9daec2e33c7962fd059eff2385f177756135c7613e40e I0704 22:25:32.397664 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-7fcb656854-6x5tq/router_current.log with fingerprint=38df73597697e04b5f149a96f2c61e909290e92bbf560ba673b2d38b85d9af0a I0704 22:25:32.397731 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-7fcb656854-h4mfx/router_current.log with fingerprint=d4cf0176f9945367d73ab78af99b4a97556df0683d9565554612d5bed50bc277 I0704 22:25:32.397740 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-5rs7n/serve-healthcheck-canary_current.log with fingerprint=4850b8513ba6c753062328febb7045a1db7073c60a3c911dfb7f50b76634b143 I0704 22:25:32.397747 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-qnm4r/serve-healthcheck-canary_current.log with fingerprint=2a8f4149c67fdddb9a81a97f7941adae940384748f0b19f24ce4280e788e9d31 I0704 22:25:32.397753 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-rztzs/serve-healthcheck-canary_current.log with fingerprint=b889d7b0ee453d34d971ddeb6475c028dab80d7b84ee3c2580d7ff326b399eef I0704 22:25:32.397773 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 449.460269ms to process 9 records W0704 22:25:32.953722 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0704 22:25:33.954012 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0704 22:25:34.425086 1 tasks_processing.go:74] worker 2 stopped. I0704 22:25:34.425200 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=51b7b4664abec5f9cc2bc7b8f8dff30b9dad66a43da669a7953f24534d4455fc I0704 22:25:34.425246 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0704 22:25:34.425271 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=664f21254f916eb41f4dac1cd74ac3a845fa6f99ab49962f331a61e3b1d49b40 I0704 22:25:34.425281 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0704 22:25:34.425304 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=3f799db6c040c90d4689c13e5668ddf0c3b08ea8dc4b93bec42927bb4bcd319e I0704 22:25:34.425321 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0704 22:25:34.425341 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=dc3232255e35d312406e4c3330dd343cc19b52be96283442d4a9212e4f28eecd I0704 22:25:34.425370 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=b75c3ed751913a02708bda754ad428be0bb8d942215ffdc9c6732b8d93849e5a I0704 22:25:34.425393 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=7fcd62dc88cfac4c963ca5134558c0a0d9e92ad61b9c97a4565d544f5f96268b I0704 22:25:34.425424 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=fe5e9ee95435f0293a7f95e257bdfe3f2b94fb12f93dec2452497d5d769f0856 I0704 22:25:34.425705 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=832205e6424370d9c543355d48766f28cc4f2e447f7a1d5d605b9bf053a20b0d I0704 22:25:34.425727 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=d896a8f786d4b6aa0349634d7316c0c23967304d1e0d837ae0b75972874662e8 I0704 22:25:34.425736 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0704 22:25:34.425754 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=68c123d6530ea496e9c2ae38d39026b4a1f2478aef68dfb419612a6353a1f5d4 I0704 22:25:34.425773 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0704 22:25:34.425790 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=c5f16c898029098e32a5991fd4f80fc77342964077bd2f75640f8820319ba1b7 I0704 22:25:34.425803 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0704 22:25:34.425824 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=0315b1f7c2cd58bb046a0ccaedc9e1f5017ab349db42c067e372786c9b933983 I0704 22:25:34.425835 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0704 22:25:34.425855 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=c218665cc41008e1135e99ca99fd96a82f391ec9b40c280ce4c544491fb1659f I0704 22:25:34.425970 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=e8e3ae02fec7d472f6198ea60a5065834661560a1d1e28499fb24519e0b2cde4 I0704 22:25:34.425981 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0704 22:25:34.425989 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0704 22:25:34.426011 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0704 22:25:34.426033 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=1c60e28150fd88656b844cd1076303d33e231a7f8ca189ce09b9b3450aa0baa9 I0704 22:25:34.426058 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=c20adb7fc5ec6544e812c8fb1a7d7af482ff21fd8febe1c25d6a762b2c58f2e8 I0704 22:25:34.426067 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0704 22:25:34.426083 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=f42fa40c6d35376b447fb67eadf32218eb7ff95d71655c9f4f73b4c5d80eb2ca I0704 22:25:34.426092 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0704 22:25:34.426110 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=df5661462281132a61b0a7da83e5f97ee1e6d994db1574db29eca6a050dc7ea5 I0704 22:25:34.426120 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0704 22:25:34.426133 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=56482b965c40f835d2c0fdc178fae42cf090048d9deea7659ad0216d178b5759 I0704 22:25:34.426150 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=214d411a6f8f6fcfa16a53c8c5dafb5fecff331f2474c8889308d7bb12bb2ad9 I0704 22:25:34.426166 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=be4f70e1e98c5ab392cb4f70f07d3db3e8d02139e49a51e2c21af0401315aea5 I0704 22:25:34.426188 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=61dc43b9d3247ffb0bda5f741293a8e35039f21e56fa11f2fd6a9ec6943c277c I0704 22:25:34.426198 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0704 22:25:34.426234 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=6c98990449662a6dbfc3d1eca3aa83a6e00e5556a360a5502ffbb751e974543c I0704 22:25:34.426252 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0704 22:25:34.426262 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0704 22:25:34.426298 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.493491783s to process 39 records W0704 22:25:34.953857 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0704 22:25:35.953357 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0704 22:25:36.951874 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0704 22:25:36.951899 1 tasks_processing.go:74] worker 19 stopped. E0704 22:25:36.951927 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0704 22:25:36.951942 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0704 22:25:36.951956 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.027779255s to process 1 records I0704 22:25:44.964468 1 tasks_processing.go:74] worker 22 stopped. I0704 22:25:44.964507 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0704 22:25:44.964539 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 13.04064981s to process 1 records I0704 22:25:45.733239 1 tasks_processing.go:74] worker 20 stopped. I0704 22:25:45.733574 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=6ddacac9eb5b9955e6f8dfad8ed1cbb9942811e254c44c5d0b6fe262c0680087 I0704 22:25:45.733592 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.807933186s to process 1 records E0704 22:25:45.733669 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.81s with: function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0704 22:25:45.734776 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "tsdb_status" failed with an error, function "metrics" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0704 22:25:45.734791 1 periodic.go:209] Running workloads gatherer I0704 22:25:45.734807 1 tasks_processing.go:45] number of workers: 2 I0704 22:25:45.734817 1 tasks_processing.go:69] worker 1 listening for tasks. I0704 22:25:45.734824 1 tasks_processing.go:71] worker 1 working on workload_info task. I0704 22:25:45.734829 1 tasks_processing.go:69] worker 0 listening for tasks. I0704 22:25:45.734839 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0704 22:25:45.743287 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-215.ec2.internal... I0704 22:25:45.743297 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-20.ec2.internal... I0704 22:25:45.743316 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-53.ec2.internal... I0704 22:25:45.766383 1 tasks_processing.go:74] worker 0 stopped. I0704 22:25:45.766403 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 31.530577ms to process 0 records I0704 22:25:50.905618 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 5s I0704 22:25:50.935321 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0704 22:25:50.946901 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (13ms) I0704 22:25:50.971884 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (25ms) I0704 22:25:51.018838 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (47ms) I0704 22:25:51.038923 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (20ms) I0704 22:25:51.051430 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (12ms) I0704 22:25:51.063144 1 gather_workloads_info.go:387] No image sha256:95418d0ba6fc0aa96647d97cf6530967f7d7ae97a82ce7f7e41c8e44310b0b72 (12ms) I0704 22:25:51.076133 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (13ms) I0704 22:25:51.089114 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (13ms) I0704 22:25:51.100387 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (11ms) I0704 22:25:51.112007 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (12ms) I0704 22:25:51.123455 1 gather_workloads_info.go:387] No image sha256:101993f6e65d6fda8b90c583b718b720809182961f69bc777295f47e6df4d7ce (11ms) I0704 22:25:51.145674 1 gather_workloads_info.go:387] No image sha256:90bdf9e678fe18e35401e34e1cb0912d822e9cb32b9e41cc545870540e9a87f2 (22ms) I0704 22:25:51.245927 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (100ms) I0704 22:25:51.346558 1 gather_workloads_info.go:387] No image sha256:7bd3e88d6d3f878a89203df590b684c5a77bf08df4717451d3197ab74683a80e (101ms) I0704 22:25:51.445430 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (99ms) I0704 22:25:51.546389 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (101ms) I0704 22:25:51.645716 1 gather_workloads_info.go:387] No image sha256:1477f3287cc75352305274f248f781c1b7c2a8696f4f7bc450c45a4a5f5b10ea (99ms) I0704 22:25:51.747096 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (101ms) I0704 22:25:51.846188 1 gather_workloads_info.go:387] No image sha256:0092c6ff139f8698c91349d507eb3182bc5d6c4ae040f672e347ff504accdea2 (99ms) I0704 22:25:51.946657 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (100ms) I0704 22:25:52.047019 1 gather_workloads_info.go:398] Found image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (100ms) I0704 22:25:52.146557 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (99ms) I0704 22:25:52.246581 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0704 22:25:52.346476 1 gather_workloads_info.go:387] No image sha256:6e5df2aab9ce759e8f957341a11c087c233307e3c8f8760d21270efc19ccd3ce (100ms) I0704 22:25:52.446751 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (100ms) I0704 22:25:52.549235 1 gather_workloads_info.go:387] No image sha256:513b1166e376feca472fdd5174f6abf0abf02925d6a21b7b4d89a2ff2fc67bed (102ms) I0704 22:25:52.646511 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (97ms) I0704 22:25:52.746212 1 gather_workloads_info.go:387] No image sha256:0af73ad7a47c9ff53269ae2bc5163bf9e389085220c34e4bd42c92c4e7a85f6d (100ms) I0704 22:25:52.846404 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0704 22:25:52.945539 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (99ms) I0704 22:25:53.046955 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (101ms) I0704 22:25:53.147364 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (100ms) I0704 22:25:53.246589 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (99ms) I0704 22:25:53.346395 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (100ms) I0704 22:25:53.447121 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (101ms) I0704 22:25:53.546050 1 gather_workloads_info.go:398] Found image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (99ms) I0704 22:25:53.645631 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (100ms) I0704 22:25:53.746279 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (101ms) I0704 22:25:53.846187 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0704 22:25:53.946256 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (100ms) I0704 22:25:54.046127 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (100ms) I0704 22:25:54.146109 1 gather_workloads_info.go:387] No image sha256:b99613494e9d6a4a8207323872d63cc02b830fc80a8bf972e62c64381aa4cd17 (100ms) I0704 22:25:54.247011 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (101ms) I0704 22:25:54.345709 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (99ms) I0704 22:25:54.446629 1 gather_workloads_info.go:387] No image sha256:74713675d580a47a56b628469c4c05c41e114e0b935960d718ff0cf7e72f613d (101ms) I0704 22:25:54.547994 1 gather_workloads_info.go:387] No image sha256:1e65f9f5490fbc930d2dce96f03ddcb28990b839d7dc011ef5de5bec6c5095bf (101ms) I0704 22:25:54.645694 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (98ms) I0704 22:25:54.645730 1 tasks_processing.go:74] worker 1 stopped. I0704 22:25:54.646015 1 recorder.go:75] Recording config/workload_info with fingerprint=38490dbfb589366671aada09973515c01c893b2c694e1045b0d6f44d4f7486a2 I0704 22:25:54.646031 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.910898448s to process 1 records I0704 22:25:54.646044 1 periodic.go:256] Periodic gather workloads completed in 8.911s I0704 22:25:54.646053 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0704 22:25:54.646058 1 periodic.go:209] Running conditional gatherer I0704 22:25:54.654279 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0704 22:25:54.817804 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer E0704 22:25:54.820176 1 conditional_gatherer.go:324] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp 172.30.94.209:9091: connect: connection refused I0704 22:25:54.820256 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0704 22:25:54.829295 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0704 22:25:54.829309 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 22:25:54.829316 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 22:25:54.829319 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 22:25:54.829323 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 22:25:54.829327 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 22:25:54.829329 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 22:25:54.829332 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 22:25:54.829335 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 22:25:54.829339 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0704 22:25:54.829343 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0704 22:25:54.829355 1 tasks_processing.go:45] number of workers: 3 I0704 22:25:54.829365 1 tasks_processing.go:69] worker 2 listening for tasks. I0704 22:25:54.829368 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0704 22:25:54.829377 1 tasks_processing.go:69] worker 0 listening for tasks. I0704 22:25:54.829388 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0704 22:25:54.829393 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0704 22:25:54.829395 1 tasks_processing.go:69] worker 1 listening for tasks. I0704 22:25:54.829407 1 tasks_processing.go:74] worker 1 stopped. I0704 22:25:54.829445 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=9fafdcbf373f1f367f1d69551ea59114592cce1681189be0f3dde774e0530661 I0704 22:25:54.829457 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 883ns to process 1 records I0704 22:25:54.829490 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0704 22:25:54.829498 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.399µs to process 1 records I0704 22:25:54.829503 1 tasks_processing.go:74] worker 0 stopped. I0704 22:25:54.829662 1 tasks_processing.go:74] worker 2 stopped. I0704 22:25:54.829674 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 261.037µs to process 0 records I0704 22:25:54.829689 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0704 22:25:54.829704 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0704 22:25:54.860230 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=3a4236035024bc1994153b6013a910c221f9b4151a5168a37882e63e7f31325b I0704 22:25:54.860364 1 diskrecorder.go:70] Writing 120 records to /var/lib/insights-operator/insights-2026-07-04-222554.tar.gz I0704 22:25:54.868203 1 diskrecorder.go:51] Wrote 120 records to disk in 7ms I0704 22:25:54.868234 1 periodic.go:278] Gathering cluster info every 2h0m0s I0704 22:25:54.868248 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0704 22:27:01.924258 1 diskrecorder.go:223] Found files to send: insights-2026-07-04-222554.tar.gz I0704 22:27:01.924289 1 insightsuploader.go:150] Checking archives to upload periodically every 15m43.607656078s I0704 22:27:01.924298 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0704 22:27:01.938299 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0704 22:27:02.063769 1 requests.go:87] Successfully reported id=2026-07-04T22:27:01Z x-rh-insights-request-id=5ebfd699a49c4d1589753e2149cf309f, wrote=57445 I0704 22:27:02.063874 1 insightsuploader.go:187] Uploaded report successfully in 139.570762ms I0704 22:27:02.063892 1 controller.go:128] Initializing last reported time to 2026-07-04T22:27:01Z I0704 22:27:02.063969 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0704 22:27:02.063982 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0704 22:27:02.063992 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0704 22:27:02.071064 1 controller.go:512] The operator is healthy I0704 22:27:31.943194 1 controller.go:512] The operator is healthy I0704 22:28:04.459289 1 insightsreport.go:137] Pulling report from smart-proxy I0704 22:28:04.459312 1 insightsreport.go:149] Retrieving report I0704 22:28:04.468841 1 requests.go:110] Retrieving report for cluster: ec009880-3840-403f-9dec-a3e6184b2194 I0704 22:28:04.468855 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/ec009880-3840-403f-9dec-a3e6184b2194/reports I0704 22:28:04.471633 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/ec009880-3840-403f-9dec-a3e6184b2194/reports I0704 22:28:04.651029 1 insightsreport.go:184] Report retrieved I0704 22:28:04.697642 1 insightsreport.go:239] Report retrieved correctly I0704 22:29:31.945644 1 controller.go:512] The operator is healthy I0704 22:30:31.914146 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0704 22:30:31.926004 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0704 22:30:31.926023 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0704 22:30:31.931819 1 secretconfigobserver.go:119] support secret does not exist I0704 22:31:31.947028 1 controller.go:512] The operator is healthy I0704 22:33:31.950863 1 controller.go:512] The operator is healthy I0704 22:35:31.908404 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 22:35:31.914428 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 22:35:31.932514 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0704 22:35:31.939400 1 controller.go:512] The operator is healthy I0704 22:35:31.940124 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0704 22:35:31.940141 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0704 22:35:31.945309 1 secretconfigobserver.go:119] support secret does not exist I0704 22:37:31.945548 1 controller.go:512] The operator is healthy I0704 22:39:31.945414 1 controller.go:512] The operator is healthy I0704 22:40:31.946064 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0704 22:40:31.952923 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0704 22:40:31.952947 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0704 22:40:31.958503 1 secretconfigobserver.go:119] support secret does not exist I0704 22:41:31.940646 1 controller.go:512] The operator is healthy I0704 22:42:50.672724 1 insightsuploader.go:146] Nothing to report since 2026-07-04T22:27:01Z I0704 22:43:31.940774 1 controller.go:512] The operator is healthy I0704 22:45:31.904540 1 configmapobserver.go:84] configmaps "insights-config" not found I0704 22:45:31.939634 1 controller.go:512] The operator is healthy I0704 22:45:31.959559 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0704 22:45:31.965635 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0704 22:45:31.965656 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0704 22:45:31.971229 1 secretconfigobserver.go:119] support secret does not exist I0704 22:47:31.941638 1 controller.go:512] The operator is healthy