I0702 02:08:42.846167 1 cmd.go:253] Using service-serving-cert provided certificates I0702 02:08:42.846638 1 observer_polling.go:159] Starting file observer I0702 02:08:43.489291 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0702 02:08:43.489472 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0702 02:08:43.491305 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0702 02:08:43.491328 1 secure_serving.go:57] Forcing use of http/1.1 only W0702 02:08:43.491359 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0702 02:08:43.491369 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0702 02:08:43.491378 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0702 02:08:43.491382 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0702 02:08:43.491385 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0702 02:08:43.491390 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0702 02:08:43.495565 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0702 02:08:43.495691 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"1e2bc19f-f827-4de2-b9d0-e590773cc764", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0702 02:08:43.496167 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0702 02:08:43.496183 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0702 02:08:43.496220 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0702 02:08:43.496231 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0702 02:08:43.496245 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0702 02:08:43.496235 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0702 02:08:43.496532 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0702 02:08:43.497011 1 secure_serving.go:213] Serving securely on [::]:8443 I0702 02:08:43.497047 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0702 02:08:43.501506 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0702 02:08:43.501533 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0702 02:08:43.501639 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0702 02:08:43.506099 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0702 02:08:43.506137 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0702 02:08:43.509908 1 secretconfigobserver.go:119] support secret does not exist I0702 02:08:43.513775 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0702 02:08:43.518399 1 secretconfigobserver.go:119] support secret does not exist I0702 02:08:43.520956 1 recorder.go:161] Pruning old reports every 5h24m59s, max age is 288h0m0s I0702 02:08:43.527390 1 periodic.go:209] Running clusterconfig gatherer I0702 02:08:43.527396 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0702 02:08:43.527412 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0702 02:08:43.527417 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0702 02:08:43.527426 1 insightsreport.go:296] Starting report retriever I0702 02:08:43.527434 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0702 02:08:43.527444 1 tasks_processing.go:45] number of workers: 32 I0702 02:08:43.527473 1 tasks_processing.go:69] worker 31 listening for tasks. I0702 02:08:43.527478 1 tasks_processing.go:71] worker 31 working on support_secret task. I0702 02:08:43.527515 1 tasks_processing.go:69] worker 15 listening for tasks. I0702 02:08:43.527523 1 tasks_processing.go:69] worker 7 listening for tasks. I0702 02:08:43.527531 1 tasks_processing.go:69] worker 23 listening for tasks. I0702 02:08:43.527531 1 tasks_processing.go:69] worker 8 listening for tasks. I0702 02:08:43.527537 1 tasks_processing.go:69] worker 9 listening for tasks. I0702 02:08:43.527544 1 tasks_processing.go:69] worker 13 listening for tasks. I0702 02:08:43.527544 1 tasks_processing.go:69] worker 16 listening for tasks. I0702 02:08:43.527554 1 tasks_processing.go:69] worker 17 listening for tasks. I0702 02:08:43.527558 1 tasks_processing.go:71] worker 7 working on version task. I0702 02:08:43.527559 1 tasks_processing.go:69] worker 14 listening for tasks. I0702 02:08:43.527563 1 tasks_processing.go:69] worker 21 listening for tasks. I0702 02:08:43.527569 1 tasks_processing.go:69] worker 22 listening for tasks. I0702 02:08:43.527573 1 tasks_processing.go:69] worker 10 listening for tasks. I0702 02:08:43.527578 1 tasks_processing.go:69] worker 11 listening for tasks. I0702 02:08:43.527573 1 tasks_processing.go:69] worker 12 listening for tasks. I0702 02:08:43.527583 1 tasks_processing.go:69] worker 18 listening for tasks. I0702 02:08:43.527582 1 tasks_processing.go:69] worker 20 listening for tasks. I0702 02:08:43.527589 1 tasks_processing.go:69] worker 0 listening for tasks. I0702 02:08:43.527582 1 tasks_processing.go:69] worker 3 listening for tasks. I0702 02:08:43.527593 1 tasks_processing.go:69] worker 1 listening for tasks. I0702 02:08:43.527596 1 tasks_processing.go:69] worker 27 listening for tasks. I0702 02:08:43.527599 1 tasks_processing.go:69] worker 19 listening for tasks. I0702 02:08:43.527601 1 tasks_processing.go:69] worker 5 listening for tasks. I0702 02:08:43.527601 1 tasks_processing.go:69] worker 2 listening for tasks. I0702 02:08:43.527608 1 tasks_processing.go:69] worker 24 listening for tasks. I0702 02:08:43.527608 1 tasks_processing.go:69] worker 25 listening for tasks. I0702 02:08:43.527608 1 tasks_processing.go:69] worker 4 listening for tasks. I0702 02:08:43.527612 1 tasks_processing.go:69] worker 6 listening for tasks. I0702 02:08:43.527524 1 tasks_processing.go:71] worker 15 working on container_images task. I0702 02:08:43.527617 1 tasks_processing.go:69] worker 29 listening for tasks. I0702 02:08:43.527616 1 tasks_processing.go:69] worker 26 listening for tasks. I0702 02:08:43.527620 1 tasks_processing.go:69] worker 28 listening for tasks. I0702 02:08:43.527623 1 tasks_processing.go:69] worker 30 listening for tasks. I0702 02:08:43.527693 1 tasks_processing.go:71] worker 14 working on install_plans task. I0702 02:08:43.527699 1 tasks_processing.go:71] worker 19 working on lokistack task. I0702 02:08:43.527702 1 tasks_processing.go:71] worker 21 working on jaegers task. I0702 02:08:43.527719 1 tasks_processing.go:71] worker 9 working on container_runtime_configs task. I0702 02:08:43.527774 1 tasks_processing.go:71] worker 5 working on proxies task. I0702 02:08:43.527907 1 tasks_processing.go:71] worker 22 working on machine_configs task. I0702 02:08:43.527957 1 tasks_processing.go:71] worker 13 working on dvo_metrics task. I0702 02:08:43.527968 1 tasks_processing.go:71] worker 29 working on service_accounts task. I0702 02:08:43.528062 1 tasks_processing.go:71] worker 26 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0702 02:08:43.528143 1 tasks_processing.go:71] worker 20 working on sap_pods task. I0702 02:08:43.528179 1 tasks_processing.go:71] worker 28 working on tsdb_status task. I0702 02:08:43.527696 1 tasks_processing.go:71] worker 30 working on aggregated_monitoring_cr_names task. I0702 02:08:43.527924 1 tasks_processing.go:71] worker 23 working on openstack_dataplanenodesets task. I0702 02:08:43.527963 1 tasks_processing.go:71] worker 6 working on clusterroles task. I0702 02:08:43.528081 1 tasks_processing.go:71] worker 17 working on active_alerts task. I0702 02:08:43.527698 1 tasks_processing.go:71] worker 27 working on mutating_webhook_configurations task. I0702 02:08:43.527937 1 tasks_processing.go:71] worker 4 working on openshift_logging task. I0702 02:08:43.527940 1 tasks_processing.go:71] worker 2 working on authentication task. I0702 02:08:43.527944 1 tasks_processing.go:71] worker 24 working on cluster_apiserver task. I0702 02:08:43.527948 1 tasks_processing.go:71] worker 25 working on ingress task. I0702 02:08:43.527953 1 tasks_processing.go:71] worker 16 working on config_maps task. I0702 02:08:43.528093 1 tasks_processing.go:71] worker 18 working on olm_operators task. I0702 02:08:43.528098 1 tasks_processing.go:71] worker 10 working on feature_gates task. I0702 02:08:43.528102 1 tasks_processing.go:71] worker 11 working on pod_network_connectivity_checks task. I0702 02:08:43.528105 1 tasks_processing.go:71] worker 12 working on storage_cluster task. I0702 02:08:43.528111 1 tasks_processing.go:71] worker 0 working on machine_config_pools task. I0702 02:08:43.528275 1 tasks_processing.go:71] worker 3 working on nodenetworkconfigurationpolicies task. I0702 02:08:43.528280 1 tasks_processing.go:71] worker 1 working on metrics task. I0702 02:08:43.527929 1 tasks_processing.go:71] worker 8 working on sap_datahubs task. I0702 02:08:43.531803 1 tasks_processing.go:71] worker 21 working on image task. I0702 02:08:43.531816 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 4.083934ms to process 0 records E0702 02:08:43.532832 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.195.46:9091: connect: connection refused I0702 02:08:43.532854 1 tasks_processing.go:71] worker 1 working on schedulers task. E0702 02:08:43.532894 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.195.46:9091: connect: connection refused E0702 02:08:43.532906 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.195.46:9091: connect: connection refused I0702 02:08:43.532933 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 2.606734ms to process 0 records E0702 02:08:43.532947 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.195.46:9091: connect: connection refused I0702 02:08:43.532955 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 4.724094ms to process 0 records I0702 02:08:43.532974 1 tasks_processing.go:71] worker 28 working on machine_healthchecks task. I0702 02:08:43.536965 1 tasks_processing.go:71] worker 20 working on machine_sets task. I0702 02:08:43.536990 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 8.802438ms to process 0 records E0702 02:08:43.537003 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0702 02:08:43.537009 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 9.496523ms to process 0 records I0702 02:08:43.537015 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 8.66715ms to process 0 records I0702 02:08:43.537022 1 tasks_processing.go:71] worker 23 working on cost_management_metrics_configs task. I0702 02:08:43.537135 1 tasks_processing.go:71] worker 31 working on validating_webhook_configurations task. I0702 02:08:43.537316 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0702 02:08:43.537342 1 controller.go:326] The initial operator extension status is healthy I0702 02:08:43.537354 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0702 02:08:43.537358 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0702 02:08:43.537362 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0702 02:08:43.537377 1 controller.go:489] The operator is still being initialized I0702 02:08:43.537385 1 controller.go:512] The operator is healthy I0702 02:08:43.537461 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0702 02:08:43.537513 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0702 02:08:43.537633 1 operator.go:288] started I0702 02:08:43.537655 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0702 02:08:43.547047 1 tasks_processing.go:71] worker 11 working on crds task. E0702 02:08:43.547061 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0702 02:08:43.547373 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 17.353917ms to process 0 records I0702 02:08:43.547384 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 17.30975ms to process 0 records I0702 02:08:43.547391 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 17.050402ms to process 0 records I0702 02:08:43.547398 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 18.143432ms to process 0 records I0702 02:08:43.547405 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 19.350081ms to process 0 records I0702 02:08:43.547413 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 19.457ms to process 0 records I0702 02:08:43.547437 1 tasks_processing.go:71] worker 19 working on openstack_controlplanes task. I0702 02:08:43.547719 1 tasks_processing.go:71] worker 9 working on machine_autoscalers task. I0702 02:08:43.547917 1 tasks_processing.go:71] worker 12 working on networks task. I0702 02:08:43.548091 1 tasks_processing.go:71] worker 5 working on openstack_dataplanedeployments task. I0702 02:08:43.547926 1 tasks_processing.go:71] worker 3 working on overlapping_namespace_uids task. I0702 02:08:43.547938 1 tasks_processing.go:71] worker 4 working on openshift_machine_api_events task. I0702 02:08:43.551960 1 recorder.go:75] Recording config/proxy with fingerprint=b44f18091abfc53c0d6ce16c665e24b68df1ade87898a2ce2705c7335d9b13e6 I0702 02:08:43.552028 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 20.303245ms to process 1 records I0702 02:08:43.552376 1 tasks_processing.go:71] worker 10 working on operators_pods_and_events task. I0702 02:08:43.552520 1 recorder.go:75] Recording config/featuregate with fingerprint=3b1c50dad5997330a98e3d78d27dd83b1aac913a708ae56c770329fc032973b4 I0702 02:08:43.552582 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 18.72408ms to process 1 records I0702 02:08:43.552686 1 tasks_processing.go:71] worker 2 working on ceph_cluster task. W0702 02:08:43.552995 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 02:08:43.553508 1 recorder.go:75] Recording config/authentication with fingerprint=aa2a0ce0ce142d6c39c1a32b879562cfce7143b8f46c3dd46e5c14b6ca96a93b I0702 02:08:43.553578 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 20.090525ms to process 1 records I0702 02:08:43.553636 1 tasks_processing.go:71] worker 24 working on silenced_alerts task. I0702 02:08:43.553893 1 recorder.go:75] Recording config/apiserver with fingerprint=63144dac3220583540dbda1c88d4f60a71e0bb7a4657396248ef9fab88c6b46c I0702 02:08:43.553921 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 21.670632ms to process 1 records I0702 02:08:43.553959 1 tasks_processing.go:71] worker 25 working on machines task. I0702 02:08:43.554678 1 recorder.go:75] Recording config/ingress with fingerprint=24bfa7e6e8865e2a467a03b249de6a9a221c55d60d37870980ba8ee767cab1f3 I0702 02:08:43.554710 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 21.668371ms to process 1 records I0702 02:08:43.554966 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=3083503066a3b5e1c9550a2ce787608e13373b2d7c85613acb108a99909723f2 I0702 02:08:43.555041 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=d6baf2c3c34f0ed11d662e76a6a4c5842e0247eeedf900b1a8a7c0af0233b411 I0702 02:08:43.555083 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=f301e6b9c24beccc6dfc30ef22c0ea882566b08bc5938b5795e94f3889bad737 I0702 02:08:43.555096 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 23.007663ms to process 3 records I0702 02:08:43.555106 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 23.489429ms to process 0 records I0702 02:08:43.554991 1 tasks_processing.go:71] worker 27 working on image_registries task. E0702 02:08:43.555144 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0702 02:08:43.555155 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 21.456545ms to process 0 records I0702 02:08:43.555164 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 25.188335ms to process 0 records I0702 02:08:43.555172 1 tasks_processing.go:71] worker 0 working on ingress_certificates task. I0702 02:08:43.555218 1 tasks_processing.go:71] worker 28 working on node_logs task. I0702 02:08:43.555530 1 tasks_processing.go:71] worker 8 working on operators task. I0702 02:08:43.555707 1 tasks_processing.go:71] worker 23 working on oauths task. I0702 02:08:43.555864 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 18.678143ms to process 0 records I0702 02:08:43.555919 1 tasks_processing.go:71] worker 1 working on nodes task. I0702 02:08:43.555990 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=ecfe9d2e34410f6744a0c0541911d43b089f2ee7a55671e56a123e6d3648e06c I0702 02:08:43.556008 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 23.05363ms to process 1 records I0702 02:08:43.556018 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 18.932285ms to process 0 records I0702 02:08:43.556028 1 tasks_processing.go:71] worker 20 working on storage_classes task. I0702 02:08:43.556110 1 recorder.go:75] Recording config/image with fingerprint=315991c02358ef1d0cf46a2f6b32b08c58b865728b4cac79504c193f2df94c26 I0702 02:08:43.556186 1 gather.go:177] gatherer "clusterconfig" function "image" took 24.158384ms to process 1 records I0702 02:08:43.556152 1 tasks_processing.go:71] worker 21 working on qemu_kubevirt_launcher_logs task. I0702 02:08:43.561770 1 tasks_processing.go:71] worker 19 working on nodenetworkstates task. I0702 02:08:43.561802 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 14.322883ms to process 0 records I0702 02:08:43.562095 1 tasks_processing.go:71] worker 5 working on openstack_version task. I0702 02:08:43.562101 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 13.985421ms to process 0 records I0702 02:08:43.562130 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 14.363897ms to process 0 records I0702 02:08:43.562140 1 tasks_processing.go:71] worker 9 working on pdbs task. I0702 02:08:43.567282 1 tasks_processing.go:71] worker 31 working on certificate_signing_requests task. I0702 02:08:43.567412 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=86e2e11d21d6e4cc788666ded4b711f9aac5766dc267e4dfce6cc92015e79e38 I0702 02:08:43.567499 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=dcb558c7e627e472b3e924d493202f1df131b49b9d10a0f135da3a3e90a8c298 I0702 02:08:43.567600 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=073df077c5a98e0f3bbcd1a9e989b50cf9a662313b2f94e7d480e63a1f471d07 I0702 02:08:43.567636 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=9732f5b3efc8af852cd43644482a27d537d6467e3a873434fdcd02d25f5ddb64 I0702 02:08:43.567668 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=f7097ea47f9ae10524818b2c8f40ff748d1b67a565a22abf7a04293e4ddfd6b3 I0702 02:08:43.567702 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=49cd6e4c388cbbe409476857d4501b324ca63644be00e2fa6d2b01faa8dc0285 I0702 02:08:43.567726 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=1bab4da4951f49bff05a4f03f9c97e9a174bc41c9e68ee269cb13f8a01fc64e8 I0702 02:08:43.567754 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=084af63fd28ebe07ab3461dac11f616886f29f5dd0e6fb09deeeae1e80179285 I0702 02:08:43.567783 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=751ced596e4a90f862c4d6e31682a767e3803f411e6d4aafea01705a686ebc3c I0702 02:08:43.567818 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=cca3505f418b4d113bf4ba245e19409857cdcf8ad2de47f97e56a1577bb7b6c1 I0702 02:08:43.567848 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=0bf837fdc81dadd9664334804b0fa76fc1ea1b5029a68f56946012d224651b7e I0702 02:08:43.567872 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=a5566b781cec9199c433342f106feb95de356b06b5e18af11a5a712d8c14ad49 I0702 02:08:43.567895 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=2ad4c96b3cd51c6fabb0fdc6323a906a5a4ef59015c40fc9dadae4f16c6ca25a I0702 02:08:43.567904 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 30.127496ms to process 13 records I0702 02:08:43.573140 1 tasks_processing.go:71] worker 2 working on infrastructures task. I0702 02:08:43.573165 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 20.376958ms to process 0 records I0702 02:08:43.573527 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0702 02:08:43.573538 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0702 02:08:43.573543 1 controller.go:212] Source scaController *sca.Controller is not ready I0702 02:08:43.573546 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0702 02:08:43.573549 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0702 02:08:43.573570 1 controller.go:489] The operator is still being initialized I0702 02:08:43.573578 1 controller.go:512] The operator is healthy I0702 02:08:43.576589 1 tasks_processing.go:71] worker 25 working on sap_config task. E0702 02:08:43.576600 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0702 02:08:43.576612 1 gather.go:177] gatherer "clusterconfig" function "machines" took 22.613999ms to process 0 records I0702 02:08:43.581513 1 tasks_processing.go:71] worker 23 working on image_pruners task. I0702 02:08:43.582966 1 recorder.go:75] Recording config/oauth with fingerprint=94f0010b97170ecf8ecd7b6d23eba539e509d0262b1a5751eeb897e49f5f1eef I0702 02:08:43.582990 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 25.727505ms to process 1 records I0702 02:08:43.583007 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 32.581677ms to process 0 records I0702 02:08:43.583426 1 tasks_processing.go:71] worker 4 working on monitoring_persistent_volumes task. I0702 02:08:43.583599 1 tasks_processing.go:74] worker 7 stopped. I0702 02:08:43.584074 1 gather_logs.go:145] no pods in namespace were found I0702 02:08:43.584517 1 recorder.go:75] Recording config/version with fingerprint=29376cd1d706f0e6ed4cd9161886cbee6d6373443309cb2478b3c834dcc570d3 I0702 02:08:43.585189 1 recorder.go:75] Recording config/id with fingerprint=6d3ed58630cd6ff943529256f4a48ed8c42354c26bf61edd0ff3d6213ae67fc7 I0702 02:08:43.585265 1 gather.go:177] gatherer "clusterconfig" function "version" took 54.519463ms to process 2 records I0702 02:08:43.585735 1 recorder.go:75] Recording config/network with fingerprint=05eedd313cbb74e0f9e6078a1affb2412fd9e08f99b39f4b6f993d0d1a1d85d4 I0702 02:08:43.585819 1 gather.go:177] gatherer "clusterconfig" function "networks" took 34.468138ms to process 1 records I0702 02:08:43.585865 1 tasks_processing.go:74] worker 12 stopped. I0702 02:08:43.586713 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=4550b17f0c36b955c0ee0c6c4e23c3f64b893d77852c9c2a50f52c331a01be95 I0702 02:08:43.587028 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=d4e68c115bb9dd43e339baca7212679d569278f2bb318269219063e6223f928b I0702 02:08:43.587096 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 54.340649ms to process 2 records I0702 02:08:43.587155 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 21.03096ms to process 0 records I0702 02:08:43.587240 1 tasks_processing.go:74] worker 20 stopped. I0702 02:08:43.587253 1 tasks_processing.go:74] worker 19 stopped. I0702 02:08:43.587173 1 tasks_processing.go:74] worker 6 stopped. I0702 02:08:43.587937 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=ec3cea0c97be87c5f22dc778d776c7abbf259f5cd9f06c58753c4d66a92b9281 I0702 02:08:43.587971 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=fd651e1c5403d124961ff592555748374ecb7f6c793e342fea400eb91388e180 I0702 02:08:43.587981 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 26.898567ms to process 2 records I0702 02:08:43.588051 1 recorder.go:75] Recording config/olm_operators with fingerprint=8612a331c9d1921a3bbf0ed317307ac215efe0acf803e3bc7ee8f0a198ed257c I0702 02:08:43.588063 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 54.32953ms to process 1 records I0702 02:08:43.588070 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 27.932767ms to process 0 records I0702 02:08:43.588076 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 28.892138ms to process 0 records I0702 02:08:43.588082 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 22.068355ms to process 0 records I0702 02:08:43.588084 1 tasks_processing.go:74] worker 18 stopped. I0702 02:08:43.588189 1 tasks_processing.go:74] worker 28 stopped. I0702 02:08:43.588244 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=142d5b69de1126a4b312ec35269891049bb3e9fcea6fdfd111a9befd9b03f83a I0702 02:08:43.588266 1 tasks_processing.go:74] worker 21 stopped. I0702 02:08:43.588271 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=2cb409a7f10f7762b310ab7c829f41223930091213387448eecf9bd0a0c7b032 I0702 02:08:43.588280 1 tasks_processing.go:74] worker 9 stopped. I0702 02:08:43.588295 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=489b8165070b293383f9bfc59c20700525e947b349fd0ef82fbf539a30e27ffb I0702 02:08:43.588317 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=b80abf38c2715a0b775452e9576d70821346af032a5003d5c54fd3f4ae82e251 I0702 02:08:43.588351 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=33804a7accaded90789a958a2295915427e07399d9bdcde027fd07eca52bf557 I0702 02:08:43.588379 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=3508d5f7a12e7bf665ead42ff023b8380afccf8cfcfe0dc4855b1f7fc6f64109 I0702 02:08:43.588459 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=2e2c886f883ec99ce4e510fd82101376544b3d2f4369f4aea82bce7e971edf3b I0702 02:08:43.588520 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=a3b963999543cae66823d8091e8d94650ca9351afbfaf4642fcebb9c96b2c822 I0702 02:08:43.588577 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=65b7875fd14a5a9841cf4409f67a1f33d31c0780066587b6fcae48c2baaccecd I0702 02:08:43.588632 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=c3db07f2451de85775571c50822412fa51d63faa1a6d9bf1e427bdc3d0e16aa0 I0702 02:08:43.588688 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=53c53508c18d66cfdaca1c21882172bd5aae108571f057c49ec6f3b7f52ffc9b I0702 02:08:43.588719 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 24.467521ms to process 11 records I0702 02:08:43.588202 1 tasks_processing.go:74] worker 5 stopped. I0702 02:08:43.588939 1 tasks_processing.go:74] worker 11 stopped. I0702 02:08:43.589513 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=e04bcfa562cf95287b7cdecc7923529106c6c76aed3e4ae263946d5ccd55ae77 I0702 02:08:43.589853 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=d0e976d02420653cb8b5b083262f93c526d95a30a385d2ea8c9669ee7deddd04 I0702 02:08:43.589867 1 gather.go:177] gatherer "clusterconfig" function "crds" took 39.950191ms to process 2 records I0702 02:08:43.589879 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0702 02:08:43.589885 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 34.188939ms to process 1 records I0702 02:08:43.589903 1 recorder.go:75] Recording config/alerts with fingerprint=43f3858c368d47c6d351c9a8270790143d77e18f11d61eb864d2d7945fe2471d I0702 02:08:43.589912 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 59.543745ms to process 1 records I0702 02:08:43.589953 1 tasks_processing.go:74] worker 24 stopped. I0702 02:08:43.589971 1 tasks_processing.go:74] worker 17 stopped. I0702 02:08:43.590005 1 tasks_processing.go:74] worker 27 stopped. I0702 02:08:43.590367 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=bedccf91d5b7eb45ec0efb591336e600d35ce08c0d7315a5db1346d5407d5d27 I0702 02:08:43.590379 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 33.042513ms to process 1 records I0702 02:08:43.590388 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 60.300205ms to process 0 records I0702 02:08:43.590394 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 12.428129ms to process 0 records I0702 02:08:43.590404 1 tasks_processing.go:74] worker 30 stopped. I0702 02:08:43.590414 1 tasks_processing.go:74] worker 25 stopped. I0702 02:08:43.590477 1 tasks_processing.go:74] worker 1 stopped. I0702 02:08:43.590680 1 recorder.go:75] Recording config/node/ip-10-0-0-79.ec2.internal with fingerprint=c1c57f9a25b9f8ac004573656d77628c21055df6dfabd4f60b5ac393a6466767 I0702 02:08:43.590749 1 recorder.go:75] Recording config/node/ip-10-0-1-114.ec2.internal with fingerprint=e0d670404c4955abfd14a9dabc47f70a9838a0c006b7ed371d79383d3e2cc5ed I0702 02:08:43.590842 1 recorder.go:75] Recording config/node/ip-10-0-2-221.ec2.internal with fingerprint=25798c43dd6ce265faf046bb605fb79482e1130a7c39660f1aba98e443fce0b3 I0702 02:08:43.590852 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 33.292926ms to process 3 records I0702 02:08:43.590870 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0702 02:08:43.590877 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 41.403209ms to process 1 records I0702 02:08:43.590946 1 tasks_processing.go:74] worker 3 stopped. I0702 02:08:43.591021 1 tasks_processing.go:74] worker 2 stopped. I0702 02:08:43.591564 1 recorder.go:75] Recording config/infrastructure with fingerprint=b8e9db542a5225116c2430545ac0d7ad275c122278ff3a4593a812ef73888147 I0702 02:08:43.591579 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 16.475123ms to process 1 records I0702 02:08:43.591587 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 38.606414ms to process 0 records I0702 02:08:43.591596 1 tasks_processing.go:74] worker 10 stopped. I0702 02:08:43.591640 1 tasks_processing.go:74] worker 23 stopped. I0702 02:08:43.591755 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=3aa38e6c538ba8ef6f0bd058632b477d74b3ec32d3399afa325ee345065b5f8f I0702 02:08:43.591782 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 10.110845ms to process 1 records I0702 02:08:43.593761 1 tasks_processing.go:74] worker 31 stopped. I0702 02:08:43.593777 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 26.456477ms to process 0 records I0702 02:08:43.596507 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0702 02:08:43.596560 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0702 02:08:43.596527 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0702 02:08:43.596737 1 tasks_processing.go:74] worker 22 stopped. I0702 02:08:43.596780 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0702 02:08:43.596796 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 68.772574ms to process 1 records I0702 02:08:43.598506 1 tasks_processing.go:74] worker 4 stopped. I0702 02:08:43.598523 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 15.054066ms to process 0 records I0702 02:08:43.601689 1 base_controller.go:82] Caches are synced for ConfigController I0702 02:08:43.601701 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0702 02:08:43.602297 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0702 02:08:43.602536 1 tasks_processing.go:74] worker 15 stopped. I0702 02:08:43.604344 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-7klw4 with fingerprint=52a2aa45ff3905acbf530f2bf47e99ef469e6afeb6623c3682d295cba8d9dcca I0702 02:08:43.604433 1 recorder.go:75] Recording config/running_containers with fingerprint=6086da44f045d12568da1371043961a7b00686ef550ff0ae0f247697cb0e1749 I0702 02:08:43.604458 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 74.906857ms to process 2 records I0702 02:08:43.604840 1 tasks_processing.go:74] worker 26 stopped. I0702 02:08:43.604854 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 76.686553ms to process 0 records I0702 02:08:43.606876 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 02:08:43.607311 1 tasks_processing.go:74] worker 16 stopped. E0702 02:08:43.607326 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0702 02:08:43.607331 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0702 02:08:43.607335 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0702 02:08:43.607343 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0702 02:08:43.607387 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=f73fcd48d6f9ebbc83253a1b5f23ef59d0683465ec46e474389103c717467dfb I0702 02:08:43.607395 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0702 02:08:43.607398 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0702 02:08:43.607407 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0702 02:08:43.607444 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0702 02:08:43.607451 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0702 02:08:43.607456 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 77.890685ms to process 7 records I0702 02:08:43.612414 1 tasks_processing.go:74] worker 0 stopped. E0702 02:08:43.612428 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0702 02:08:43.612434 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2r9nbgn6pfcnkmq08cm2r044mkhpiuqc-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2r9nbgn6pfcnkmq08cm2r044mkhpiuqc-primary-cert-bundle-secret" not found I0702 02:08:43.612480 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=8b576dc77d1d3407e68f955f62654c9606d4b9ae1a6440dfdd8d449709f5f596 I0702 02:08:43.612492 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 57.228588ms to process 1 records I0702 02:08:43.638508 1 base_controller.go:82] Caches are synced for LoggingSyncer I0702 02:08:43.638524 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0702 02:08:43.654958 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0702 02:08:43.659368 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0702 02:08:43.680916 1 cluster_transfer.go:102] no available accepted cluster transfer I0702 02:08:43.680940 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0702 02:08:43.787383 1 sca.go:266] etc-pki-entitlement secret successfully created I0702 02:08:43.787404 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated W0702 02:08:44.551003 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0702 02:08:45.550879 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0702 02:08:46.014410 1 tasks_processing.go:74] worker 8 stopped. I0702 02:08:46.014530 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=1549df4be46af685aea29950cafc968b661b78ff43570e8f9b4ef6f0413ac63d I0702 02:08:46.014588 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0702 02:08:46.014615 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=c5d42566b4f361466b307ff4cacff5dab0c72cfaa864ff962258b1f400654ebe I0702 02:08:46.014625 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0702 02:08:46.014646 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=56973a368923d33dab7fc889a19a52cf674abb8fd5211e675af753204b16394a I0702 02:08:46.014662 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0702 02:08:46.014685 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=e927d2d9af274f7e212a6525e86d0e4c6469e6e309c6d26351dc76a0ac4eff6f I0702 02:08:46.014712 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=42d45f9f43cbbd4ffbae9a891f8949ed3d1fda7a63c1e093b5f18e86185f270f I0702 02:08:46.014735 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=54ca560c3e30704909261e9b12ec323cbae44e1a50969fef672903b6d60d509b I0702 02:08:46.014765 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=8655d4853a12712f928ea81648c94f48d8b13c6009b54553c880bca5baae37b7 I0702 02:08:46.015027 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=4f7562e82073127459d26eabb52bc290911e4fb0355808261c5c3a6cd09003a4 I0702 02:08:46.015045 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=b2fea8c716d8bb89aa0b4a9125158ee1c1b0102f6b719678e2aa5003237f4fc0 I0702 02:08:46.015054 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0702 02:08:46.015069 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=af20e22a6bdbc32b55719492ad22af27c4b5c453f5fa087052552553c1a0e5ad I0702 02:08:46.015077 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0702 02:08:46.015093 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=1a06572b6ebc518af34635ae10753f1d193c273b5acb41fc20c88e5bb4c69c47 I0702 02:08:46.015103 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0702 02:08:46.015138 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=74ded08ca1733e98ca68a2610a9d1a0ce7dc2671279de12061439d61f4b7f08b I0702 02:08:46.015147 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0702 02:08:46.015166 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=dce0b71d3b43c8f2d9a8951b92dec7e32e373a51564964981d8f38abfdc1bb3a I0702 02:08:46.015290 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=ae37496beb1a805bc88295bbdf9656a2c58abf77a517af71defc1e5c4c001748 I0702 02:08:46.015300 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0702 02:08:46.015307 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0702 02:08:46.015328 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0702 02:08:46.015351 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=12bdb4ee13154b95e236751710e02bf2853faca917bc1cb735c3d089bde26154 I0702 02:08:46.015375 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=d9f69fee9c2c023f65d52ff6e5ade59ef512cf871dcee2caddc9b54347cb2dc9 I0702 02:08:46.015384 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0702 02:08:46.015399 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=55487114cbb36a8d88c3a2067302a82aef5c61920cb99e263d2ab4c3dce4825a I0702 02:08:46.015408 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0702 02:08:46.015424 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=63de5faf5cf4198c4cb3160ab825135955897edda18a1284997e8055303e322d I0702 02:08:46.015447 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0702 02:08:46.015464 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=375f7236516b98d9982615438bf2630e0c10817f119a9c6f59d7c51a199bed72 I0702 02:08:46.015479 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=65b7ab7fb8adaf1674cdf362ed7146adcd349b512597b7d0d9705c26dc1016a3 I0702 02:08:46.015495 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=d5a56978555aebfada11411a69eb126f53b184af59ffbf7177ac59a7fdc5c5dc I0702 02:08:46.015516 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=8e3e4ad590964180b7cab40ee860d9dd508b4e74165ef85fa92a4ba252bf910e I0702 02:08:46.015525 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0702 02:08:46.015552 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=80d559c0986b109bd316b550fbfc1e191b40b74b103bb531b455d8d9371238eb I0702 02:08:46.015570 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0702 02:08:46.015579 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0702 02:08:46.015588 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.45885218s to process 39 records W0702 02:08:46.550820 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0702 02:08:47.550183 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0702 02:08:48.548779 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0702 02:08:48.548805 1 tasks_processing.go:74] worker 13 stopped. E0702 02:08:48.548817 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0702 02:08:48.548828 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0702 02:08:48.548837 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.020827122s to process 1 records I0702 02:08:56.560683 1 tasks_processing.go:74] worker 14 stopped. I0702 02:08:56.560731 1 recorder.go:75] Recording config/installplans with fingerprint=e3c18441424aaa81f54ea9723d194eacb5824eb47178047bd893de11825bc8a2 I0702 02:08:56.560748 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 13.032965823s to process 1 records I0702 02:08:57.334223 1 tasks_processing.go:74] worker 29 stopped. I0702 02:08:57.334534 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=ae29e7659fb4cd0f04bcc181913d3f8b69c20a612f3ed6bf2f2479314a8cc58f I0702 02:08:57.334553 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.806233936s to process 1 records E0702 02:08:57.334614 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.807s with: function \"metrics\" failed with an error, function \"tsdb_status\" failed with an error, function \"support_secret\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0702 02:08:57.335715 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "metrics" failed with an error, function "tsdb_status" failed with an error, function "support_secret" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0702 02:08:57.335729 1 periodic.go:209] Running workloads gatherer I0702 02:08:57.335744 1 tasks_processing.go:45] number of workers: 2 I0702 02:08:57.335753 1 tasks_processing.go:69] worker 1 listening for tasks. I0702 02:08:57.335756 1 tasks_processing.go:71] worker 1 working on workload_info task. I0702 02:08:57.335766 1 tasks_processing.go:69] worker 0 listening for tasks. I0702 02:08:57.335840 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0702 02:08:57.342139 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-221.ec2.internal... I0702 02:08:57.342202 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-79.ec2.internal... I0702 02:08:57.342220 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-114.ec2.internal... I0702 02:08:57.365169 1 tasks_processing.go:74] worker 0 stopped. I0702 02:08:57.365188 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 29.311249ms to process 0 records I0702 02:09:03.034608 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 6s I0702 02:09:03.064203 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0702 02:09:03.073615 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (11ms) I0702 02:09:03.083619 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (10ms) I0702 02:09:03.092869 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (9ms) I0702 02:09:03.103267 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (10ms) I0702 02:09:03.112078 1 gather_workloads_info.go:387] No image sha256:6e5df2aab9ce759e8f957341a11c087c233307e3c8f8760d21270efc19ccd3ce (9ms) I0702 02:09:03.123788 1 gather_workloads_info.go:387] No image sha256:74713675d580a47a56b628469c4c05c41e114e0b935960d718ff0cf7e72f613d (12ms) I0702 02:09:03.133673 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (10ms) I0702 02:09:03.143165 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (9ms) I0702 02:09:03.152655 1 gather_workloads_info.go:387] No image sha256:88437b35173fe2e8a4d9f148ec17c754bee4df3f723b3a5676955eef4f363d31 (9ms) I0702 02:09:03.161820 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (9ms) I0702 02:09:03.172888 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (11ms) I0702 02:09:03.274183 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (101ms) I0702 02:09:03.374146 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0702 02:09:03.472995 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (99ms) I0702 02:09:03.574278 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (101ms) I0702 02:09:03.673624 1 gather_workloads_info.go:387] No image sha256:101993f6e65d6fda8b90c583b718b720809182961f69bc777295f47e6df4d7ce (99ms) I0702 02:09:03.773474 1 gather_workloads_info.go:387] No image sha256:95418d0ba6fc0aa96647d97cf6530967f7d7ae97a82ce7f7e41c8e44310b0b72 (100ms) I0702 02:09:03.873871 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (100ms) I0702 02:09:03.973877 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (100ms) I0702 02:09:04.074767 1 gather_workloads_info.go:387] No image sha256:0af73ad7a47c9ff53269ae2bc5163bf9e389085220c34e4bd42c92c4e7a85f6d (101ms) I0702 02:09:04.174094 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (99ms) I0702 02:09:04.275650 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (102ms) I0702 02:09:04.379982 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (104ms) I0702 02:09:04.474209 1 gather_workloads_info.go:387] No image sha256:90bdf9e678fe18e35401e34e1cb0912d822e9cb32b9e41cc545870540e9a87f2 (94ms) I0702 02:09:04.573648 1 gather_workloads_info.go:387] No image sha256:0092c6ff139f8698c91349d507eb3182bc5d6c4ae040f672e347ff504accdea2 (99ms) I0702 02:09:04.672888 1 gather_workloads_info.go:387] No image sha256:513b1166e376feca472fdd5174f6abf0abf02925d6a21b7b4d89a2ff2fc67bed (99ms) I0702 02:09:04.774102 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (101ms) I0702 02:09:04.874010 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (100ms) I0702 02:09:04.982280 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (108ms) I0702 02:09:05.074047 1 gather_workloads_info.go:387] No image sha256:7bd3e88d6d3f878a89203df590b684c5a77bf08df4717451d3197ab74683a80e (92ms) I0702 02:09:05.175879 1 gather_workloads_info.go:398] Found image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (102ms) I0702 02:09:05.274150 1 gather_workloads_info.go:398] Found image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (98ms) I0702 02:09:05.384100 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (110ms) I0702 02:09:05.473146 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (89ms) I0702 02:09:05.574077 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (101ms) I0702 02:09:05.673761 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (100ms) I0702 02:09:05.773341 1 gather_workloads_info.go:387] No image sha256:1e65f9f5490fbc930d2dce96f03ddcb28990b839d7dc011ef5de5bec6c5095bf (100ms) I0702 02:09:05.873062 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (100ms) I0702 02:09:05.973436 1 gather_workloads_info.go:387] No image sha256:b99613494e9d6a4a8207323872d63cc02b830fc80a8bf972e62c64381aa4cd17 (100ms) I0702 02:09:06.073011 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0702 02:09:06.173449 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (100ms) I0702 02:09:06.273498 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0702 02:09:06.373509 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0702 02:09:06.473578 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (100ms) I0702 02:09:06.573522 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0702 02:09:06.673664 1 gather_workloads_info.go:387] No image sha256:1477f3287cc75352305274f248f781c1b7c2a8696f4f7bc450c45a4a5f5b10ea (100ms) I0702 02:09:06.772964 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (99ms) I0702 02:09:06.875799 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (103ms) I0702 02:09:06.972983 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (97ms) I0702 02:09:06.973022 1 tasks_processing.go:74] worker 1 stopped. E0702 02:09:06.973034 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: received unexpected status code 500 Internal Server Error from https://10.128.0.15:8443/gather_runtime_info I0702 02:09:06.973353 1 recorder.go:75] Recording config/workload_info with fingerprint=d332fdd7772e472bbf8b31d62ccc5a9a2cd0b2204967e734531a580ffe37309c I0702 02:09:06.973373 1 gather.go:177] gatherer "workloads" function "workload_info" took 9.637257921s to process 1 records E0702 02:09:06.973409 1 periodic.go:247] "Unhandled Error" err="workloads failed after 9.637s with: function \"workload_info\" failed with an error" I0702 02:09:06.974514 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0702 02:09:06.974528 1 periodic.go:209] Running conditional gatherer I0702 02:09:06.981266 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0702 02:09:07.184981 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0702 02:09:08.266257 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0702 02:09:08.272587 1 conditional_gatherer.go:392] cluster version is '4.20.8' I0702 02:09:08.272613 1 tasks_processing.go:45] number of workers: 3 I0702 02:09:08.272620 1 tasks_processing.go:69] worker 2 listening for tasks. I0702 02:09:08.272624 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0702 02:09:08.272690 1 tasks_processing.go:69] worker 1 listening for tasks. I0702 02:09:08.272704 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0702 02:09:08.272706 1 tasks_processing.go:71] worker 1 working on remote_configuration task. I0702 02:09:08.272708 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0702 02:09:08.272714 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 965ns to process 1 records I0702 02:09:08.272719 1 tasks_processing.go:69] worker 0 listening for tasks. I0702 02:09:08.272729 1 tasks_processing.go:74] worker 0 stopped. I0702 02:09:08.272753 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0702 02:09:08.272763 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.402µs to process 1 records I0702 02:09:08.272769 1 tasks_processing.go:74] worker 1 stopped. I0702 02:09:08.273010 1 tasks_processing.go:74] worker 2 stopped. I0702 02:09:08.273025 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 284.36µs to process 0 records I0702 02:09:08.273050 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0702 02:09:08.273067 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0702 02:09:08.297212 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=e9075c61319f36769984f27ca609615243dd7c7276d91895c62e545dacb52c56 I0702 02:09:08.297391 1 diskrecorder.go:70] Writing 111 records to /var/lib/insights-operator/insights-2026-07-02-020908.tar.gz I0702 02:09:08.304877 1 diskrecorder.go:51] Wrote 111 records to disk in 7ms I0702 02:09:08.304909 1 periodic.go:278] Gathering cluster info every 2h0m0s I0702 02:09:08.304926 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0702 02:10:13.527990 1 diskrecorder.go:223] Found files to send: insights-2026-07-02-020908.tar.gz I0702 02:10:13.528038 1 insightsuploader.go:150] Checking archives to upload periodically every 15m28.291714863s I0702 02:10:13.528047 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0702 02:10:13.543384 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0702 02:10:13.720105 1 requests.go:87] Successfully reported id=2026-07-02T02:10:13Z x-rh-insights-request-id=f65c5893653f46518bba0888ba077d23, wrote=49784 I0702 02:10:13.720252 1 insightsuploader.go:187] Uploaded report successfully in 192.196002ms I0702 02:10:13.720288 1 controller.go:128] Initializing last reported time to 2026-07-02T02:10:13Z I0702 02:10:13.720308 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0702 02:10:13.720315 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0702 02:10:13.720323 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0702 02:10:13.725939 1 controller.go:512] The operator is healthy I0702 02:10:43.543608 1 controller.go:512] The operator is healthy I0702 02:11:16.932349 1 insightsreport.go:137] Pulling report from smart-proxy I0702 02:11:16.932378 1 insightsreport.go:149] Retrieving report I0702 02:11:16.939786 1 requests.go:110] Retrieving report for cluster: 1f1b1aa0-98eb-4f1b-b150-ad61cdc00ef0 I0702 02:11:16.939827 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/1f1b1aa0-98eb-4f1b-b150-ad61cdc00ef0/reports I0702 02:11:16.942824 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/1f1b1aa0-98eb-4f1b-b150-ad61cdc00ef0/reports I0702 02:11:17.103613 1 insightsreport.go:184] Report retrieved I0702 02:11:17.141687 1 insightsreport.go:239] Report retrieved correctly I0702 02:12:43.543946 1 controller.go:512] The operator is healthy I0702 02:13:43.519284 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0702 02:13:43.526208 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0702 02:13:43.526230 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0702 02:13:43.530583 1 secretconfigobserver.go:119] support secret does not exist I0702 02:14:43.543631 1 controller.go:512] The operator is healthy I0702 02:16:43.721403 1 controller.go:512] The operator is healthy I0702 02:18:43.517964 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 02:18:43.531099 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0702 02:18:43.536401 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0702 02:18:43.536422 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0702 02:18:43.540777 1 secretconfigobserver.go:119] support secret does not exist I0702 02:18:43.543825 1 controller.go:512] The operator is healthy I0702 02:20:43.544563 1 controller.go:512] The operator is healthy I0702 02:22:43.542947 1 controller.go:512] The operator is healthy I0702 02:23:43.541556 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0702 02:23:43.547234 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0702 02:23:43.547256 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0702 02:23:43.551167 1 secretconfigobserver.go:119] support secret does not exist I0702 02:24:43.543332 1 controller.go:512] The operator is healthy I0702 02:25:47.013455 1 insightsuploader.go:146] Nothing to report since 2026-07-02T02:10:13Z I0702 02:26:43.543746 1 controller.go:512] The operator is healthy I0702 02:28:43.518325 1 configmapobserver.go:84] configmaps "insights-config" not found I0702 02:28:43.543670 1 controller.go:512] The operator is healthy I0702 02:28:43.551378 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0702 02:28:43.555973 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0702 02:28:43.555999 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0702 02:28:43.559866 1 secretconfigobserver.go:119] support secret does not exist I0702 02:30:43.543379 1 controller.go:512] The operator is healthy