I0319 14:01:38.714830 1 cmd.go:253] Using service-serving-cert provided certificates I0319 14:01:38.715280 1 observer_polling.go:159] Starting file observer I0319 14:01:39.071517 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0319 14:01:39.071829 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0319 14:01:39.072774 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0319 14:01:39.073141 1 secure_serving.go:57] Forcing use of http/1.1 only W0319 14:01:39.073172 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0319 14:01:39.073181 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0319 14:01:39.073186 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0319 14:01:39.073190 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0319 14:01:39.073192 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0319 14:01:39.073195 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0319 14:01:39.076067 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0319 14:01:39.076078 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"1adc1221-c25b-4bd5-b4f1-330bbfe8caf2", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0319 14:01:39.078315 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0319 14:01:39.078323 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0319 14:01:39.078332 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0319 14:01:39.078341 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0319 14:01:39.078349 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0319 14:01:39.078337 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0319 14:01:39.078681 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0319 14:01:39.079114 1 secure_serving.go:213] Serving securely on [::]:8443 I0319 14:01:39.079156 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0319 14:01:39.080935 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0319 14:01:39.080967 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0319 14:01:39.081160 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0319 14:01:39.085211 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0319 14:01:39.085232 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0319 14:01:39.088983 1 secretconfigobserver.go:119] support secret does not exist I0319 14:01:39.093587 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0319 14:01:39.096911 1 secretconfigobserver.go:119] support secret does not exist I0319 14:01:39.099048 1 recorder.go:161] Pruning old reports every 5h34m7s, max age is 288h0m0s I0319 14:01:39.102753 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0319 14:01:39.102774 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0319 14:01:39.102782 1 periodic.go:209] Running clusterconfig gatherer I0319 14:01:39.102824 1 tasks_processing.go:45] number of workers: 32 I0319 14:01:39.102858 1 tasks_processing.go:69] worker 4 listening for tasks. I0319 14:01:39.102865 1 tasks_processing.go:69] worker 7 listening for tasks. I0319 14:01:39.102871 1 tasks_processing.go:69] worker 8 listening for tasks. I0319 14:01:39.102873 1 tasks_processing.go:69] worker 5 listening for tasks. I0319 14:01:39.102882 1 tasks_processing.go:69] worker 6 listening for tasks. I0319 14:01:39.102883 1 tasks_processing.go:69] worker 2 listening for tasks. I0319 14:01:39.102888 1 tasks_processing.go:69] worker 3 listening for tasks. I0319 14:01:39.102889 1 tasks_processing.go:69] worker 1 listening for tasks. I0319 14:01:39.102779 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0319 14:01:39.102897 1 tasks_processing.go:69] worker 0 listening for tasks. I0319 14:01:39.102898 1 tasks_processing.go:69] worker 9 listening for tasks. I0319 14:01:39.102898 1 tasks_processing.go:69] worker 10 listening for tasks. I0319 14:01:39.102950 1 tasks_processing.go:69] worker 12 listening for tasks. I0319 14:01:39.102967 1 tasks_processing.go:69] worker 13 listening for tasks. I0319 14:01:39.103057 1 insightsreport.go:296] Starting report retriever I0319 14:01:39.103198 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0319 14:01:39.103299 1 tasks_processing.go:69] worker 26 listening for tasks. I0319 14:01:39.103351 1 tasks_processing.go:71] worker 26 working on active_alerts task. I0319 14:01:39.103377 1 tasks_processing.go:71] worker 6 working on aggregated_monitoring_cr_names task. I0319 14:01:39.103442 1 tasks_processing.go:69] worker 28 listening for tasks. I0319 14:01:39.103396 1 tasks_processing.go:69] worker 20 listening for tasks. I0319 14:01:39.103472 1 tasks_processing.go:69] worker 16 listening for tasks. I0319 14:01:39.103311 1 tasks_processing.go:69] worker 11 listening for tasks. I0319 14:01:39.103323 1 tasks_processing.go:69] worker 27 listening for tasks. I0319 14:01:39.103324 1 tasks_processing.go:69] worker 15 listening for tasks. I0319 14:01:39.103326 1 tasks_processing.go:69] worker 14 listening for tasks. I0319 14:01:39.103494 1 tasks_processing.go:69] worker 29 listening for tasks. I0319 14:01:39.103547 1 tasks_processing.go:71] worker 2 working on support_secret task. I0319 14:01:39.103624 1 tasks_processing.go:69] worker 17 listening for tasks. I0319 14:01:39.103668 1 tasks_processing.go:71] worker 7 working on image task. I0319 14:01:39.103779 1 tasks_processing.go:69] worker 19 listening for tasks. I0319 14:01:39.103821 1 tasks_processing.go:71] worker 8 working on sap_datahubs task. I0319 14:01:39.104036 1 tasks_processing.go:71] worker 3 working on machine_autoscalers task. I0319 14:01:39.104145 1 tasks_processing.go:71] worker 5 working on ingress_certificates task. I0319 14:01:39.103633 1 tasks_processing.go:69] worker 30 listening for tasks. I0319 14:01:39.103643 1 tasks_processing.go:69] worker 31 listening for tasks. I0319 14:01:39.103664 1 tasks_processing.go:71] worker 4 working on openshift_logging task. I0319 14:01:39.104419 1 tasks_processing.go:71] worker 1 working on service_accounts task. I0319 14:01:39.104543 1 tasks_processing.go:69] worker 24 listening for tasks. I0319 14:01:39.104597 1 tasks_processing.go:69] worker 25 listening for tasks. I0319 14:01:39.103302 1 tasks_processing.go:71] worker 13 working on tsdb_status task. I0319 14:01:39.103677 1 tasks_processing.go:69] worker 18 listening for tasks. I0319 14:01:39.104463 1 tasks_processing.go:69] worker 23 listening for tasks. I0319 14:01:39.104455 1 tasks_processing.go:69] worker 22 listening for tasks. I0319 14:01:39.105175 1 tasks_processing.go:71] worker 12 working on metrics task. I0319 14:01:39.105211 1 tasks_processing.go:71] worker 0 working on storage_cluster task. I0319 14:01:39.105271 1 tasks_processing.go:71] worker 22 working on feature_gates task. I0319 14:01:39.105383 1 tasks_processing.go:71] worker 9 working on container_runtime_configs task. I0319 14:01:39.105174 1 tasks_processing.go:71] worker 10 working on machine_healthchecks task. I0319 14:01:39.105315 1 tasks_processing.go:71] worker 29 working on storage_classes task. I0319 14:01:39.104445 1 tasks_processing.go:69] worker 21 listening for tasks. I0319 14:01:39.105320 1 tasks_processing.go:71] worker 31 working on machine_sets task. I0319 14:01:39.105324 1 tasks_processing.go:71] worker 19 working on openstack_dataplanedeployments task. I0319 14:01:39.105329 1 tasks_processing.go:71] worker 30 working on openstack_version task. I0319 14:01:39.105336 1 tasks_processing.go:71] worker 17 working on node_logs task. I0319 14:01:39.105340 1 tasks_processing.go:71] worker 11 working on schedulers task. I0319 14:01:39.105346 1 tasks_processing.go:71] worker 28 working on openstack_controlplanes task. I0319 14:01:39.105347 1 tasks_processing.go:71] worker 20 working on overlapping_namespace_uids task. I0319 14:01:39.107216 1 tasks_processing.go:71] worker 3 working on cost_management_metrics_configs task. I0319 14:01:39.105353 1 tasks_processing.go:71] worker 16 working on cluster_apiserver task. I0319 14:01:39.105359 1 tasks_processing.go:71] worker 25 working on qemu_kubevirt_launcher_logs task. I0319 14:01:39.105361 1 tasks_processing.go:71] worker 15 working on mutating_webhook_configurations task. I0319 14:01:39.105367 1 tasks_processing.go:71] worker 24 working on authentication task. I0319 14:01:39.105370 1 tasks_processing.go:71] worker 27 working on certificate_signing_requests task. I0319 14:01:39.105376 1 tasks_processing.go:71] worker 14 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0319 14:01:39.105376 1 tasks_processing.go:71] worker 18 working on nodenetworkconfigurationpolicies task. I0319 14:01:39.105383 1 tasks_processing.go:71] worker 23 working on image_pruners task. I0319 14:01:39.106026 1 tasks_processing.go:71] worker 21 working on install_plans task. I0319 14:01:39.107237 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 3.128441ms to process 0 records I0319 14:01:39.107358 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 3.395708ms to process 0 records I0319 14:01:39.107369 1 tasks_processing.go:71] worker 8 working on olm_operators task. I0319 14:01:39.108125 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 3.734441ms to process 0 records I0319 14:01:39.108157 1 tasks_processing.go:71] worker 4 working on networks task. I0319 14:01:39.108546 1 tasks_processing.go:71] worker 0 working on operators_pods_and_events task. I0319 14:01:39.108556 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 3.306942ms to process 0 records I0319 14:01:39.108799 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0319 14:01:39.108815 1 controller.go:326] The initial operator extension status is healthy I0319 14:01:39.108822 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0319 14:01:39.108827 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0319 14:01:39.108829 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0319 14:01:39.108845 1 controller.go:489] The operator is still being initialized I0319 14:01:39.108853 1 controller.go:512] The operator is healthy I0319 14:01:39.108936 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0319 14:01:39.109072 1 operator.go:288] started E0319 14:01:39.109075 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.128.50:9091: connect: connection refused E0319 14:01:39.109077 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.128.50:9091: connect: connection refused I0319 14:01:39.109093 1 tasks_processing.go:71] worker 13 working on container_images task. I0319 14:01:39.109100 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0319 14:01:39.109110 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s E0319 14:01:39.109131 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.128.50:9091: connect: connection refused I0319 14:01:39.109151 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 4.437175ms to process 0 records E0319 14:01:39.109161 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.128.50:9091: connect: connection refused I0319 14:01:39.109175 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 3.903714ms to process 0 records I0319 14:01:39.109264 1 tasks_processing.go:71] worker 12 working on dvo_metrics task. I0319 14:01:39.111418 1 tasks_processing.go:71] worker 19 working on lokistack task. I0319 14:01:39.111436 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 5.297735ms to process 0 records I0319 14:01:39.111637 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 5.37759ms to process 0 records I0319 14:01:39.111665 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 6.005618ms to process 0 records E0319 14:01:39.111677 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0319 14:01:39.111686 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 7.89034ms to process 0 records I0319 14:01:39.111693 1 tasks_processing.go:71] worker 31 working on crds task. I0319 14:01:39.111707 1 tasks_processing.go:71] worker 2 working on jaegers task. I0319 14:01:39.111734 1 tasks_processing.go:71] worker 30 working on validating_webhook_configurations task. I0319 14:01:39.111696 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 5.23251ms to process 0 records E0319 14:01:39.111889 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0319 14:01:39.111900 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 6.150341ms to process 0 records I0319 14:01:39.111922 1 tasks_processing.go:71] worker 10 working on sap_pods task. I0319 14:01:39.111742 1 tasks_processing.go:71] worker 9 working on ceph_cluster task. I0319 14:01:39.112060 1 recorder.go:75] Recording config/image with fingerprint=3ee3d651768a45c9801b9caa4ce397d95df3ba880ebe73e3b8108dcf1c2f2588 I0319 14:01:39.112080 1 gather.go:177] gatherer "clusterconfig" function "image" took 8.031549ms to process 1 records I0319 14:01:39.112265 1 tasks_processing.go:71] worker 22 working on oauths task. I0319 14:01:39.112345 1 recorder.go:75] Recording config/featuregate with fingerprint=3b690ba30516952f78f26ee9921ea87fb92ceeef1ece058584a719443bf2fce5 I0319 14:01:39.112391 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 6.753504ms to process 1 records I0319 14:01:39.112271 1 tasks_processing.go:71] worker 7 working on machines task. I0319 14:01:39.117300 1 tasks_processing.go:71] worker 18 working on sap_config task. I0319 14:01:39.117723 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 9.91004ms to process 0 records I0319 14:01:39.119450 1 tasks_processing.go:71] worker 28 working on proxies task. I0319 14:01:39.119952 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 12.343927ms to process 0 records I0319 14:01:39.120640 1 tasks_processing.go:71] worker 15 working on ingress task. I0319 14:01:39.120974 1 gather_logs.go:145] no pods in namespace were found I0319 14:01:39.122138 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=e84c663363226e3c115436d60f6e9a19387c828683f34ac6a3982e4a9783d246 I0319 14:01:39.122999 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 13.298408ms to process 1 records I0319 14:01:39.123187 1 tasks_processing.go:71] worker 11 working on infrastructures task. I0319 14:01:39.123455 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=b89e78ec8cf75930049410c746f115d0bf97db039e0e3a40ea83e9a2889362fe I0319 14:01:39.123478 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 14.092671ms to process 1 records I0319 14:01:39.123490 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 13.516183ms to process 0 records I0319 14:01:39.123764 1 recorder.go:75] Recording config/apiserver with fingerprint=2e4741691ae6bcba149bf61bbdd43bb7d11db8d837c64dfbd7501057368cc21a I0319 14:01:39.123788 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 14.397023ms to process 1 records I0319 14:01:39.123803 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 15.56778ms to process 0 records I0319 14:01:39.123898 1 tasks_processing.go:71] worker 16 working on openshift_machine_api_events task. I0319 14:01:39.123985 1 tasks_processing.go:71] worker 17 working on pod_network_connectivity_checks task. I0319 14:01:39.124026 1 tasks_processing.go:71] worker 29 working on config_maps task. I0319 14:01:39.124010 1 tasks_processing.go:71] worker 3 working on version task. I0319 14:01:39.123967 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=328e2f1f01e3489a3d45319fbbce545d4c7a1604a6bbb977d14fb4f0a07ca578 I0319 14:01:39.124723 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=05094e47ac4d7443d8de23e1aae7168826286bb871d426f16f862ece2cd7936b I0319 14:01:39.125073 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 16.136071ms to process 2 records I0319 14:01:39.125090 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 14.812049ms to process 0 records I0319 14:01:39.125121 1 tasks_processing.go:71] worker 25 working on machine_config_pools task. I0319 14:01:39.125421 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=d8c64a007ece0b84931fa85ef35a816796421651f9f4cd8ff3d53c3db89e7b2d I0319 14:01:39.125495 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 15.157629ms to process 1 records I0319 14:01:39.125459 1 tasks_processing.go:71] worker 23 working on nodenetworkstates task. I0319 14:01:39.125688 1 tasks_processing.go:71] worker 24 working on machine_configs task. I0319 14:01:39.126020 1 recorder.go:75] Recording config/authentication with fingerprint=54ce3c9fe1358660147cf525f9697da4853c4c97c6f2cc4b35d79ab7d63d19b5 I0319 14:01:39.126077 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 16.014706ms to process 1 records I0319 14:01:39.126101 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0319 14:01:39.126136 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 18.899304ms to process 1 records I0319 14:01:39.126150 1 tasks_processing.go:71] worker 20 working on clusterroles task. I0319 14:01:39.159955 1 tasks_processing.go:71] worker 28 working on silenced_alerts task. I0319 14:01:39.160042 1 recorder.go:75] Recording config/proxy with fingerprint=d8297ed6a36cb8d55d0b518c75cefae634a733f1c76b0ba80949130974658177 I0319 14:01:39.160091 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 40.443164ms to process 1 records I0319 14:01:39.160168 1 tasks_processing.go:71] worker 4 working on nodes task. I0319 14:01:39.160444 1 recorder.go:75] Recording config/network with fingerprint=59f6d003abb90b0542cd9b2084508d4ca542e392e0acd1b53d4a065ab4c241cc I0319 14:01:39.160508 1 gather.go:177] gatherer "clusterconfig" function "networks" took 51.997413ms to process 1 records I0319 14:01:39.160590 1 tasks_processing.go:71] worker 30 working on openstack_dataplanenodesets task. I0319 14:01:39.160752 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=8eee43ffb7a7051c76cae5ed10e2fc33449ac3e7ba545e22b7d739b33aed8cb0 I0319 14:01:39.160855 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=fcc6fdb8662eb154695cec9cd36e640aef45e250ab6a34f5a31761182aa77e9b I0319 14:01:39.161044 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=3e27f995ee93783e518dd88633abee2f5b0cefa89ce057edafa59452d017c694 I0319 14:01:39.161164 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=b31ccbc922d4547db245fad971505d0f2747bb9300404058ff9ebfa1a29ba494 I0319 14:01:39.161213 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=bf1e476852da61e6c7e9e0009102c13b6ef9b450e9114993c958b86aaf8cbb0f I0319 14:01:39.161230 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 48.810426ms to process 5 records I0319 14:01:39.167429 1 tasks_processing.go:71] worker 16 working on monitoring_persistent_volumes task. I0319 14:01:39.167476 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 43.446365ms to process 0 records I0319 14:01:39.167787 1 tasks_processing.go:71] worker 9 working on operators task. I0319 14:01:39.168200 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 55.77331ms to process 0 records E0319 14:01:39.168702 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0319 14:01:39.168742 1 gather.go:177] gatherer "clusterconfig" function "machines" took 55.566538ms to process 0 records I0319 14:01:39.168824 1 tasks_processing.go:71] worker 7 working on pdbs task. I0319 14:01:39.168792 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 56.412613ms to process 0 records I0319 14:01:39.169028 1 tasks_processing.go:71] worker 2 working on image_registries task. I0319 14:01:39.169223 1 recorder.go:75] Recording config/ingress with fingerprint=2a8153bdcfdc5cd05849a37bf1f04cce8beb1f98c19ae66cb26337755382a2fd I0319 14:01:39.169352 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 47.528002ms to process 1 records I0319 14:01:39.169387 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 56.766228ms to process 0 records I0319 14:01:39.169398 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 43.160271ms to process 0 records E0319 14:01:39.169406 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0319 14:01:39.169413 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 44.26342ms to process 0 records I0319 14:01:39.169419 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 50.68201ms to process 0 records I0319 14:01:39.169425 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 56.554774ms to process 0 records I0319 14:01:39.169430 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 43.275155ms to process 0 records I0319 14:01:39.169510 1 tasks_processing.go:74] worker 17 stopped. I0319 14:01:39.169533 1 tasks_processing.go:74] worker 10 stopped. I0319 14:01:39.169541 1 tasks_processing.go:74] worker 18 stopped. I0319 14:01:39.169549 1 tasks_processing.go:74] worker 19 stopped. I0319 14:01:39.169281 1 tasks_processing.go:74] worker 15 stopped. I0319 14:01:39.169553 1 tasks_processing.go:74] worker 23 stopped. I0319 14:01:39.169559 1 tasks_processing.go:74] worker 25 stopped. I0319 14:01:39.169683 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0319 14:01:39.169726 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0319 14:01:39.169734 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0319 14:01:39.169739 1 controller.go:212] Source scaController *sca.Controller is not ready I0319 14:01:39.169742 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0319 14:01:39.169765 1 controller.go:489] The operator is still being initialized I0319 14:01:39.169781 1 controller.go:512] The operator is healthy I0319 14:01:39.169695 1 tasks_processing.go:74] worker 11 stopped. I0319 14:01:39.170234 1 recorder.go:75] Recording config/infrastructure with fingerprint=72fde780956d5bc17dbffe77083a97d4f6a3d18d510af1061b785a59f6bc1a54 I0319 14:01:39.170277 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 45.689888ms to process 1 records I0319 14:01:39.170302 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 65.80474ms to process 0 records I0319 14:01:39.170330 1 tasks_processing.go:74] worker 6 stopped. I0319 14:01:39.170366 1 recorder.go:75] Recording config/olm_operators with fingerprint=bbcaea63b88b15776312f311de0893ed685137119f892be517b6b6c1e6b75478 I0319 14:01:39.170383 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 62.05693ms to process 1 records I0319 14:01:39.170402 1 tasks_processing.go:74] worker 8 stopped. I0319 14:01:39.172634 1 tasks_processing.go:74] worker 30 stopped. I0319 14:01:39.172649 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 12.029124ms to process 0 records I0319 14:01:39.172750 1 tasks_processing.go:74] worker 3 stopped. I0319 14:01:39.173193 1 recorder.go:75] Recording config/version with fingerprint=d468d4c65cdf854b915d69a027c8b8ba51cd850e603b5b52e75a016e3143f848 I0319 14:01:39.173208 1 recorder.go:75] Recording config/id with fingerprint=a59834817b68a2f6e7e51a106d5b0a568fef722ef30ebde2a7f4aae65deaeba3 I0319 14:01:39.173214 1 gather.go:177] gatherer "clusterconfig" function "version" took 48.078936ms to process 2 records I0319 14:01:39.173534 1 tasks_processing.go:74] worker 20 stopped. I0319 14:01:39.174404 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=fb90084edda22b0246294e10d45989c6efd0b7991e416e25350a49e448bc82b9 I0319 14:01:39.174632 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=5e09928c17c8f0f39a7390e2057ea8cae561b9bafdaf82f2b73102a81dc09f69 I0319 14:01:39.174652 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 47.368543ms to process 2 records I0319 14:01:39.174820 1 tasks_processing.go:74] worker 22 stopped. I0319 14:01:39.175346 1 recorder.go:75] Recording config/oauth with fingerprint=81f9ac1122a15f62d78321f9479452cdc873487a80edb138f02e2eda995a810e I0319 14:01:39.175410 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 62.534752ms to process 1 records I0319 14:01:39.176228 1 recorder.go:75] Recording config/node/ip-10-0-0-28.ec2.internal with fingerprint=9c2b72facaa615310f00cbbd874d88205981dddd5d2ecc0b9e43753cccd533bb I0319 14:01:39.176411 1 recorder.go:75] Recording config/node/ip-10-0-1-137.ec2.internal with fingerprint=4680de57937cab2f132d84f4b94524028e00b4882f0fb9188bb760b8e62a4e23 I0319 14:01:39.176538 1 recorder.go:75] Recording config/node/ip-10-0-2-97.ec2.internal with fingerprint=39838ca861c4af78daadb17c553497483d9ec41074803e90251a37935e494d3e I0319 14:01:39.176551 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 15.196856ms to process 3 records I0319 14:01:39.176563 1 tasks_processing.go:74] worker 4 stopped. I0319 14:01:39.178632 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0319 14:01:39.178636 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController W0319 14:01:39.179349 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0319 14:01:39.178651 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0319 14:01:39.181300 1 base_controller.go:82] Caches are synced for ConfigController I0319 14:01:39.181315 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0319 14:01:39.183090 1 tasks_processing.go:74] worker 16 stopped. I0319 14:01:39.183109 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 15.641348ms to process 0 records I0319 14:01:39.183870 1 tasks_processing.go:74] worker 27 stopped. I0319 14:01:39.183892 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 76.299601ms to process 0 records I0319 14:01:39.183983 1 tasks_processing.go:74] worker 2 stopped. I0319 14:01:39.184667 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=5e39ec7cd9e94cc407d5fbaf899b14cabfaa9c3f8395751869e69bbcc6f5c418 I0319 14:01:39.184697 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 14.935181ms to process 1 records I0319 14:01:39.184797 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=f5b1675ab84c9eaf9ffaa7004003c2a02e057b532933002ff2eb6f1e5079b82b I0319 14:01:39.184816 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=846b14f02b83e7928b18334dad4caa59057910b1eeaaa29675ae07e85b0e91fb I0319 14:01:39.184832 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=9877a01263be023fdc78a569f87494eb926eba7e3945fa585f1f17ccd86a69bd I0319 14:01:39.184846 1 tasks_processing.go:74] worker 7 stopped. I0319 14:01:39.184852 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=3e1f8cd3e30a85d4cc3b22823efdc63c0964b32bef68e247a9d5d8185021789e I0319 14:01:39.184877 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=8d2e1901c8b6decc5d6bdefa4d6d282382aed6f0c0a3beee215f2add0ae94659 I0319 14:01:39.184904 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=2f2c28788143242d0911543c32f28d6771197c10f0e666a80dcf833da59dd07d I0319 14:01:39.184931 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=89731d5aef3a86db2a83e14036f327be5884087ee1ac564ea99157604de9c753 I0319 14:01:39.184954 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=a138d0d76dbb7e6054cdd4f3c28a199dd185130787e1b9ac790e2eedb46d0453 I0319 14:01:39.184978 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=43b5dd1b793c27d62d54f6140e4690f55608daed1e094311cf35dbd03bafc2e3 I0319 14:01:39.185004 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=5071ffe0e5ad157636d63f0fe2eec01010d3085df9904c08c8346759082c368f I0319 14:01:39.185027 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=4e0cb1a1e4159644731e6bf479fcb86c673e3e8360c0b190eaf369d95da1c437 I0319 14:01:39.185036 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 15.367847ms to process 11 records I0319 14:01:39.185165 1 tasks_processing.go:74] worker 31 stopped. I0319 14:01:39.185610 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=34b1510b599747e888a6e370562f672895307611f6776b30668993331a0aed4a I0319 14:01:39.185854 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=f8b6b23fe980ec324baa3ca70ce2cfe57f6dcbe21c923feb05627cd669066e02 I0319 14:01:39.185867 1 gather.go:177] gatherer "clusterconfig" function "crds" took 72.855523ms to process 2 records I0319 14:01:39.185878 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0319 14:01:39.185886 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 24.549748ms to process 1 records I0319 14:01:39.185914 1 recorder.go:75] Recording config/alerts with fingerprint=1296f792d7a8996be67027b929b40aaad293eab6b8ba0094266a55758b0c6788 I0319 14:01:39.185949 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 81.537396ms to process 1 records I0319 14:01:39.185950 1 tasks_processing.go:74] worker 28 stopped. I0319 14:01:39.185958 1 tasks_processing.go:74] worker 26 stopped. I0319 14:01:39.187601 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 14:01:39.190625 1 tasks_processing.go:74] worker 24 stopped. I0319 14:01:39.190665 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0319 14:01:39.190678 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 64.816746ms to process 1 records I0319 14:01:39.192347 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0319 14:01:39.193759 1 tasks_processing.go:74] worker 5 stopped. E0319 14:01:39.193773 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0319 14:01:39.193785 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2p4qu4f22mlbdnpskeg8ajj9mnqkh5tm-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2p4qu4f22mlbdnpskeg8ajj9mnqkh5tm-primary-cert-bundle-secret" not found I0319 14:01:39.193850 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=9ea662ad69b044a228de8d523f89af6b622071514cc96722bd502fc0c16a7cd9 I0319 14:01:39.193868 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 89.558303ms to process 1 records I0319 14:01:39.196650 1 tasks_processing.go:74] worker 14 stopped. I0319 14:01:39.196668 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 89.297719ms to process 0 records I0319 14:01:39.196742 1 tasks_processing.go:74] worker 13 stopped. I0319 14:01:39.198564 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-9p2jj with fingerprint=51829b38ac64e2a6b80eceecc0ff0c640139f3fade7c0fd0a0f07cebc2caf471 I0319 14:01:39.198837 1 recorder.go:75] Recording config/pod/openshift-ovn-kubernetes/ovnkube-node-cqltx with fingerprint=6b5472e9b0983e5a3a3821b596f3cb5906fd2df03dadea4848f8a1fbf24a62a8 I0319 14:01:39.198905 1 recorder.go:75] Recording config/running_containers with fingerprint=fc429d7973f922cf68fe03627121a50ae3bf5e5b4c15db690d532c57efd922b2 I0319 14:01:39.198915 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 87.630972ms to process 3 records I0319 14:01:39.201351 1 tasks_processing.go:74] worker 29 stopped. E0319 14:01:39.201412 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0319 14:01:39.201450 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0319 14:01:39.201478 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0319 14:01:39.201512 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0319 14:01:39.201573 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0319 14:01:39.201601 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0319 14:01:39.201627 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0319 14:01:39.201676 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0319 14:01:39.201759 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0319 14:01:39.201787 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0319 14:01:39.201814 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 77.268785ms to process 7 records I0319 14:01:39.210022 1 base_controller.go:82] Caches are synced for LoggingSyncer I0319 14:01:39.210034 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0319 14:01:39.210342 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-55957fc79-sn68r pod in namespace openshift-ingress (previous: false). I0319 14:01:39.220986 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-75d6856fff-p6qk6 pod in namespace openshift-ingress (previous: false). I0319 14:01:39.231420 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-75d6856fff-qc88t pod in namespace openshift-ingress (previous: false). I0319 14:01:39.233235 1 cluster_transfer.go:102] no available accepted cluster transfer I0319 14:01:39.233261 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0319 14:01:39.235052 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0319 14:01:39.237901 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0319 14:01:39.241723 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-4xt2m pod in namespace openshift-ingress-canary (previous: false). I0319 14:01:39.251113 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-5mbrw pod in namespace openshift-ingress-canary (previous: false). I0319 14:01:39.330841 1 sca.go:266] etc-pki-entitlement secret successfully created I0319 14:01:39.330861 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0319 14:01:39.376277 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-dkpk2 pod in namespace openshift-ingress-canary (previous: false). I0319 14:01:39.576159 1 tasks_processing.go:74] worker 0 stopped. I0319 14:01:39.576289 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=f5bafa2caf4a0d05c0e66a249b61d4aff58276bf7fddac3ec203f2a3b4ce88cd I0319 14:01:39.576394 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=34badef7e4b2f544d2db1602e738301df35cb7ef99e1d88c2e86171525a8f56b I0319 14:01:39.576436 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=52cb0b7f57e7bc167c388a2d8bb6ef7b7211560d830593d7fb3fe0ded0d8082b I0319 14:01:39.576471 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-55957fc79-sn68r/router_current.log with fingerprint=bc86ef575eb2df55b81e6a79b151af91d7c6eabc66f5fcec0f11abfe1d73d351 I0319 14:01:39.576525 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-75d6856fff-p6qk6/router_current.log with fingerprint=421fd2ea7b0b59d785638fa3c555a58ddd4b543bfb35f5f2b0f916bf5b6dceb9 I0319 14:01:39.576570 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-75d6856fff-qc88t/router_current.log with fingerprint=2f3ea98eb538e40d4a054495257f17ab6b28da9fcb4e84b2291cc9fb637eeac8 I0319 14:01:39.576576 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-4xt2m/serve-healthcheck-canary_current.log with fingerprint=b2eab67519a233115146d22230eac9167e7536ae11414cc05b3973ccb3d07a6e I0319 14:01:39.576580 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-5mbrw/serve-healthcheck-canary_current.log with fingerprint=b786cd56a63c36687717c3959d87d5176e89295936a1280020305ea49271cd9d I0319 14:01:39.576584 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-dkpk2/serve-healthcheck-canary_current.log with fingerprint=273daa06a5f980457d4f6cb87c285688456b25e0a54fb15145b5b5543fc461e8 I0319 14:01:39.576590 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 467.569497ms to process 9 records W0319 14:01:40.176218 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0319 14:01:41.175754 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0319 14:01:41.605976 1 tasks_processing.go:74] worker 9 stopped. I0319 14:01:41.606106 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=9985ba8d2baa2716e40d71a2da4b48436870386dbd4b0f58ac352d192f6524fb I0319 14:01:41.606151 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0319 14:01:41.606172 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=acc7807fca525716deab07813e6df36f133237f12bc43bca3c39ef946cb3170a I0319 14:01:41.606181 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0319 14:01:41.606205 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=0968ec2c101dbf2e3d5d1a361492fbd19cc3dab84ff7e69cd0fe555a5ab0b24c I0319 14:01:41.606221 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0319 14:01:41.606262 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=f61d9fb5372c6856ca513d1181e69f9ce42bddf9180ffd4a092dc6a445abddc9 I0319 14:01:41.606291 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=c21b2a11648b3d49fdd7d76d0f53d55fb71c8d912244ee355a9cf3c41054fc07 I0319 14:01:41.606314 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=1a45e906ddda45d395f291c3e1d8bcaeb07a778cb0aa9b76c1b5e4852a6d53eb I0319 14:01:41.606344 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=d105e1892c5162fab18fb703d80ddaf8da2df24d1992cfbae0732ed4e7dbd762 I0319 14:01:41.606608 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=9f8916d06efc2b57f80138f13f9ce728be09f9bbf556d84ebd67dba13d3c63c2 I0319 14:01:41.606627 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=b6ceb617795a064cd75a295026c42e96758e450c8793ed0997fa83023067fb83 I0319 14:01:41.606635 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0319 14:01:41.606651 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=8cbbc732417573217da32e68de0bc499e4d014338efd525b83199df91cde0d50 I0319 14:01:41.606662 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0319 14:01:41.606678 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=380a90f3c30070dc7395a6df24ce5455c0891570017821d73f47df1e8e7fad31 I0319 14:01:41.606687 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0319 14:01:41.606705 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=ab1965aaa2b59dea06baa266c57298da720b27dfc48c38bcae7a9a214ea34396 I0319 14:01:41.606715 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0319 14:01:41.606735 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=b4d968b6c04740c93ee7d92f7f45ec6e240df8cb46cbe308180bc741a2b8893d I0319 14:01:41.606837 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=2007ab6491cc44db4e407e8759889b795b0104ef4d6bfecb294bb3b09f5372c2 I0319 14:01:41.606846 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0319 14:01:41.606853 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0319 14:01:41.606874 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0319 14:01:41.606896 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=63e6af2e815dc03eba43ba9e21e2f6fed1ff4c8aeeaa97701acd530f8732242e I0319 14:01:41.606919 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=9fd0fc5a50e4629758dc04cd2477ff525cf5746c95e7a352a0afce10dc2b79ff I0319 14:01:41.606928 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0319 14:01:41.606945 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=6e14d04daf466d46a1e722b95d8807ebce0941ff14f523fd27ec58d6cddbbb33 I0319 14:01:41.606954 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0319 14:01:41.606971 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=e572aee71bb47e40dc3ddebb3267e4f604238bf142d49022b67f3aca904c67d9 I0319 14:01:41.606980 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0319 14:01:41.606993 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=3cecd02f62620e2075c214fc47c5aa5bcbcb2c622dda5bc91dde1be5bf1ee2e3 I0319 14:01:41.607008 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=cad16f74229884850048faac6ae271200dfaa2d6c1119ad120d2b0bc4a46544e I0319 14:01:41.607024 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=c0eabc4d3dfe077a494a184d26f0991c850f69284c7af0cb5cd4411bdea12df8 I0319 14:01:41.607047 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=d484514656ea8500d7c59314fe261d8552bbaf92b86bfe3bcafd8bf60b250b1b I0319 14:01:41.607057 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0319 14:01:41.607080 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=9efbd1a99c18120020790d8f879886ab86000f266136e3ee2845055d5570d2e2 I0319 14:01:41.607097 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0319 14:01:41.607107 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0319 14:01:41.607114 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.438121959s to process 39 records W0319 14:01:42.175369 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0319 14:01:43.175332 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0319 14:01:44.173694 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0319 14:01:44.173737 1 tasks_processing.go:74] worker 12 stopped. E0319 14:01:44.173736 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0319 14:01:44.173761 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0319 14:01:44.173772 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.064439472s to process 1 records I0319 14:01:51.930558 1 tasks_processing.go:74] worker 21 stopped. I0319 14:01:51.930608 1 recorder.go:75] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0319 14:01:51.930620 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.823144338s to process 1 records I0319 14:01:52.710029 1 tasks_processing.go:74] worker 1 stopped. I0319 14:01:52.710405 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=4aff5860ec073ddfdf1bec96303c96ef7753a0d476f8315de0a2062bde3a2fd5 I0319 14:01:52.710423 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.605426236s to process 1 records E0319 14:01:52.710495 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.607s with: function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"support_secret\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error" I0319 14:01:52.711597 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "tsdb_status" failed with an error, function "metrics" failed with an error, function "support_secret" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error I0319 14:01:52.711614 1 periodic.go:209] Running workloads gatherer I0319 14:01:52.711627 1 tasks_processing.go:45] number of workers: 2 I0319 14:01:52.711633 1 tasks_processing.go:69] worker 1 listening for tasks. I0319 14:01:52.711637 1 tasks_processing.go:71] worker 1 working on workload_info task. I0319 14:01:52.711645 1 tasks_processing.go:69] worker 0 listening for tasks. I0319 14:01:52.711664 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0319 14:01:52.717225 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-28.ec2.internal... I0319 14:01:52.717236 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-137.ec2.internal... I0319 14:01:52.717263 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-97.ec2.internal... I0319 14:01:52.730781 1 tasks_processing.go:74] worker 0 stopped. I0319 14:01:52.730798 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 19.095564ms to process 0 records I0319 14:01:56.872654 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 4s I0319 14:01:56.903372 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0319 14:01:56.932457 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (30ms) I0319 14:01:56.940368 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (8ms) I0319 14:01:56.948097 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (8ms) I0319 14:01:56.955387 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (7ms) I0319 14:01:56.963140 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (8ms) I0319 14:01:56.970265 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (7ms) I0319 14:01:56.977509 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (7ms) I0319 14:01:56.984881 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (7ms) I0319 14:01:56.991895 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (7ms) I0319 14:01:56.999099 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (7ms) I0319 14:01:57.010068 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (11ms) I0319 14:01:57.110980 1 gather_workloads_info.go:387] No image sha256:0af73ad7a47c9ff53269ae2bc5163bf9e389085220c34e4bd42c92c4e7a85f6d (101ms) I0319 14:01:57.210555 1 gather_workloads_info.go:387] No image sha256:1477f3287cc75352305274f248f781c1b7c2a8696f4f7bc450c45a4a5f5b10ea (100ms) I0319 14:01:57.311563 1 gather_workloads_info.go:387] No image sha256:513b1166e376feca472fdd5174f6abf0abf02925d6a21b7b4d89a2ff2fc67bed (101ms) I0319 14:01:57.410995 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (99ms) I0319 14:01:57.511881 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (101ms) I0319 14:01:57.610966 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (99ms) I0319 14:01:57.710831 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (100ms) I0319 14:01:57.811670 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (101ms) I0319 14:01:57.911334 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (100ms) I0319 14:01:58.011592 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (100ms) I0319 14:01:58.110785 1 gather_workloads_info.go:398] Found image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (99ms) I0319 14:01:58.210558 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0319 14:01:58.311621 1 gather_workloads_info.go:387] No image sha256:7bd3e88d6d3f878a89203df590b684c5a77bf08df4717451d3197ab74683a80e (101ms) I0319 14:01:58.411151 1 gather_workloads_info.go:387] No image sha256:6e5df2aab9ce759e8f957341a11c087c233307e3c8f8760d21270efc19ccd3ce (100ms) I0319 14:01:58.511083 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (100ms) I0319 14:01:58.611196 1 gather_workloads_info.go:387] No image sha256:09e27b3a67ebeda04d3e597e0561bfb40550de78bdecb53f2f9ff3dfb1cfe7ea (100ms) I0319 14:01:58.711701 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (100ms) I0319 14:01:58.811811 1 gather_workloads_info.go:387] No image sha256:90bdf9e678fe18e35401e34e1cb0912d822e9cb32b9e41cc545870540e9a87f2 (100ms) I0319 14:01:58.911750 1 gather_workloads_info.go:387] No image sha256:1e65f9f5490fbc930d2dce96f03ddcb28990b839d7dc011ef5de5bec6c5095bf (100ms) I0319 14:01:59.011437 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0319 14:01:59.111346 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (100ms) I0319 14:01:59.211009 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (100ms) I0319 14:01:59.310492 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (99ms) I0319 14:01:59.412032 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (102ms) I0319 14:01:59.510826 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (99ms) I0319 14:01:59.610422 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (100ms) I0319 14:01:59.711187 1 gather_workloads_info.go:387] No image sha256:74713675d580a47a56b628469c4c05c41e114e0b935960d718ff0cf7e72f613d (101ms) I0319 14:01:59.811678 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (100ms) I0319 14:01:59.911229 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (100ms) I0319 14:02:00.011006 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0319 14:02:00.110996 1 gather_workloads_info.go:387] No image sha256:b99613494e9d6a4a8207323872d63cc02b830fc80a8bf972e62c64381aa4cd17 (100ms) I0319 14:02:00.210648 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0319 14:02:00.311668 1 gather_workloads_info.go:398] Found image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (101ms) I0319 14:02:00.412069 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (100ms) I0319 14:02:00.511112 1 gather_workloads_info.go:387] No image sha256:0092c6ff139f8698c91349d507eb3182bc5d6c4ae040f672e347ff504accdea2 (99ms) I0319 14:02:00.611730 1 gather_workloads_info.go:387] No image sha256:101993f6e65d6fda8b90c583b718b720809182961f69bc777295f47e6df4d7ce (101ms) I0319 14:02:00.712799 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (101ms) I0319 14:02:00.712833 1 tasks_processing.go:74] worker 1 stopped. E0319 14:02:00.712843 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: received unexpected status code 500 Internal Server Error from https://10.129.0.15:8443/gather_runtime_info I0319 14:02:00.713113 1 recorder.go:75] Recording config/workload_info with fingerprint=87cb8bd8a466520d97ba3d60e277fa23b2de73e9a44adfe263e11e114bf7a980 I0319 14:02:00.713129 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.001187617s to process 1 records E0319 14:02:00.713153 1 periodic.go:247] "Unhandled Error" err="workloads failed after 8.001s with: function \"workload_info\" failed with an error" I0319 14:02:00.714264 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0319 14:02:00.714277 1 periodic.go:209] Running conditional gatherer I0319 14:02:00.718805 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0319 14:02:00.855278 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0319 14:02:01.938949 1 conditional_gatherer.go:374] alert "KubeStatefulSetReplicasMismatch" has state "pending" I0319 14:02:01.938963 1 conditional_gatherer.go:374] alert "KubeContainerWaiting" has state "pending" I0319 14:02:01.938966 1 conditional_gatherer.go:374] alert "KubeJobFailed" has state "pending" I0319 14:02:01.938969 1 conditional_gatherer.go:374] alert "KubePdbNotEnoughHealthyPods" has state "pending" I0319 14:02:01.938978 1 conditional_gatherer.go:374] alert "Watchdog" has state "firing" I0319 14:02:01.939050 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0319 14:02:01.944613 1 conditional_gatherer.go:392] cluster version is '4.20.8' I0319 14:02:01.944648 1 tasks_processing.go:45] number of workers: 3 I0319 14:02:01.944659 1 tasks_processing.go:69] worker 2 listening for tasks. I0319 14:02:01.944663 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0319 14:02:01.944672 1 tasks_processing.go:69] worker 0 listening for tasks. I0319 14:02:01.944686 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0319 14:02:01.944692 1 tasks_processing.go:69] worker 1 listening for tasks. I0319 14:02:01.944694 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0319 14:02:01.944706 1 tasks_processing.go:74] worker 1 stopped. I0319 14:02:01.945054 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0319 14:02:01.945146 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.057µs to process 1 records I0319 14:02:01.945236 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0319 14:02:01.945297 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.85µs to process 1 records I0319 14:02:01.945336 1 tasks_processing.go:74] worker 0 stopped. I0319 14:02:01.945632 1 tasks_processing.go:74] worker 2 stopped. I0319 14:02:01.945672 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 921.19µs to process 0 records I0319 14:02:01.945718 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0319 14:02:01.945768 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0319 14:02:01.973783 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=f981a0e34bd6bba5ee8d044dcf34a0f026d61eb5e8ae47dfef90496e4ccb3e2f I0319 14:02:01.973929 1 diskrecorder.go:70] Writing 111 records to /var/lib/insights-operator/insights-2026-03-19-140201.tar.gz I0319 14:02:01.981646 1 diskrecorder.go:51] Wrote 111 records to disk in 7ms I0319 14:02:01.981681 1 periodic.go:278] Gathering cluster info every 2h0m0s I0319 14:02:01.981696 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0319 14:03:09.103348 1 diskrecorder.go:223] Found files to send: insights-2026-03-19-140201.tar.gz I0319 14:03:09.103379 1 insightsuploader.go:150] Checking archives to upload periodically every 15m40.494032105s I0319 14:03:09.103388 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0319 14:03:09.111828 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0319 14:03:09.566479 1 requests.go:87] Successfully reported id=2026-03-19T14:03:09Z x-rh-insights-request-id=a5728e9166d746fa85cf01d2b9331c63, wrote=56395 I0319 14:03:09.566596 1 insightsuploader.go:187] Uploaded report successfully in 463.199172ms I0319 14:03:09.566622 1 controller.go:128] Initializing last reported time to 2026-03-19T14:03:09Z I0319 14:03:09.566693 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0319 14:03:09.566708 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0319 14:03:09.566720 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0319 14:03:09.571307 1 controller.go:512] The operator is healthy I0319 14:03:39.115051 1 controller.go:512] The operator is healthy I0319 14:04:11.942952 1 insightsreport.go:137] Pulling report from smart-proxy I0319 14:04:11.942975 1 insightsreport.go:149] Retrieving report I0319 14:04:11.949274 1 requests.go:110] Retrieving report for cluster: 5608c07e-ea17-4aa3-bf46-947a921087f3 I0319 14:04:11.949285 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/5608c07e-ea17-4aa3-bf46-947a921087f3/reports I0319 14:04:11.953361 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/5608c07e-ea17-4aa3-bf46-947a921087f3/reports I0319 14:04:12.144070 1 insightsreport.go:184] Report retrieved I0319 14:04:12.170450 1 insightsreport.go:239] Report retrieved correctly I0319 14:05:39.114636 1 controller.go:512] The operator is healthy I0319 14:06:39.097162 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0319 14:06:39.119105 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0319 14:06:39.119124 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0319 14:06:39.142851 1 secretconfigobserver.go:119] support secret does not exist I0319 14:07:39.115203 1 controller.go:512] The operator is healthy I0319 14:09:39.114876 1 controller.go:512] The operator is healthy I0319 14:11:39.097491 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 14:11:39.113587 1 controller.go:512] The operator is healthy I0319 14:11:39.143323 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0319 14:11:39.148701 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0319 14:11:39.148722 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0319 14:11:39.151986 1 secretconfigobserver.go:119] support secret does not exist I0319 14:13:39.114654 1 controller.go:512] The operator is healthy