W0319 19:20:02.331198 1 cmd.go:257] Using insecure, self-signed certificates I0319 19:20:03.087957 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 19:20:03.088328 1 observer_polling.go:159] Starting file observer I0319 19:20:03.851256 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0319 19:20:03.851527 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0319 19:20:03.852283 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0319 19:20:03.852772 1 secure_serving.go:57] Forcing use of http/1.1 only W0319 19:20:03.852896 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0319 19:20:03.852941 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0319 19:20:03.852969 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0319 19:20:03.852998 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0319 19:20:03.853021 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0319 19:20:03.853043 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0319 19:20:03.856792 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0319 19:20:03.856849 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"a4863bf4-e4eb-48c6-949b-31c360b2bc61", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0319 19:20:03.860480 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0319 19:20:03.860499 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0319 19:20:03.860563 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0319 19:20:03.860570 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0319 19:20:03.860585 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0319 19:20:03.860586 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0319 19:20:03.860896 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-3785252497/tls.crt::/tmp/serving-cert-3785252497/tls.key" I0319 19:20:03.861237 1 secure_serving.go:213] Serving securely on [::]:8443 I0319 19:20:03.861268 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0319 19:20:03.863677 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0319 19:20:03.863705 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0319 19:20:03.863853 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0319 19:20:03.870312 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0319 19:20:03.870334 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0319 19:20:03.877061 1 secretconfigobserver.go:119] support secret does not exist I0319 19:20:03.890455 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0319 19:20:03.901287 1 secretconfigobserver.go:119] support secret does not exist I0319 19:20:03.906339 1 recorder.go:176] Pruning old reports every 7h18m56s, max age is 288h0m0s I0319 19:20:03.912352 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0319 19:20:03.912361 1 periodic.go:216] Running clusterconfig gatherer I0319 19:20:03.912375 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0319 19:20:03.912418 1 tasks_processing.go:45] number of workers: 32 I0319 19:20:03.912424 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0319 19:20:03.912441 1 insightsreport.go:296] Starting report retriever I0319 19:20:03.912448 1 tasks_processing.go:69] worker 1 listening for tasks. I0319 19:20:03.912450 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0319 19:20:03.912459 1 tasks_processing.go:71] worker 1 working on node_logs task. I0319 19:20:03.912460 1 tasks_processing.go:69] worker 16 listening for tasks. I0319 19:20:03.912468 1 tasks_processing.go:69] worker 23 listening for tasks. I0319 19:20:03.912468 1 tasks_processing.go:69] worker 24 listening for tasks. I0319 19:20:03.912477 1 tasks_processing.go:69] worker 25 listening for tasks. I0319 19:20:03.912471 1 tasks_processing.go:69] worker 2 listening for tasks. I0319 19:20:03.912478 1 tasks_processing.go:69] worker 3 listening for tasks. I0319 19:20:03.912485 1 tasks_processing.go:69] worker 4 listening for tasks. I0319 19:20:03.912491 1 tasks_processing.go:69] worker 17 listening for tasks. I0319 19:20:03.912493 1 tasks_processing.go:69] worker 5 listening for tasks. I0319 19:20:03.912493 1 tasks_processing.go:69] worker 26 listening for tasks. I0319 19:20:03.912499 1 tasks_processing.go:69] worker 6 listening for tasks. I0319 19:20:03.912500 1 tasks_processing.go:69] worker 27 listening for tasks. I0319 19:20:03.912506 1 tasks_processing.go:69] worker 28 listening for tasks. I0319 19:20:03.912506 1 tasks_processing.go:69] worker 7 listening for tasks. I0319 19:20:03.912514 1 tasks_processing.go:69] worker 21 listening for tasks. I0319 19:20:03.912517 1 tasks_processing.go:69] worker 29 listening for tasks. I0319 19:20:03.912519 1 tasks_processing.go:69] worker 20 listening for tasks. I0319 19:20:03.912524 1 tasks_processing.go:69] worker 22 listening for tasks. I0319 19:20:03.912525 1 tasks_processing.go:69] worker 30 listening for tasks. I0319 19:20:03.912450 1 tasks_processing.go:69] worker 0 listening for tasks. I0319 19:20:03.912530 1 tasks_processing.go:69] worker 18 listening for tasks. I0319 19:20:03.912532 1 tasks_processing.go:69] worker 14 listening for tasks. I0319 19:20:03.912541 1 tasks_processing.go:69] worker 12 listening for tasks. I0319 19:20:03.912544 1 tasks_processing.go:69] worker 13 listening for tasks. I0319 19:20:03.912545 1 tasks_processing.go:69] worker 19 listening for tasks. I0319 19:20:03.912549 1 tasks_processing.go:69] worker 9 listening for tasks. I0319 19:20:03.912554 1 tasks_processing.go:71] worker 2 working on networks task. I0319 19:20:03.912555 1 tasks_processing.go:69] worker 10 listening for tasks. I0319 19:20:03.912558 1 tasks_processing.go:71] worker 9 working on machines task. I0319 19:20:03.912559 1 tasks_processing.go:69] worker 11 listening for tasks. I0319 19:20:03.912563 1 tasks_processing.go:71] worker 10 working on machine_healthchecks task. I0319 19:20:03.912566 1 tasks_processing.go:71] worker 23 working on storage_cluster task. I0319 19:20:03.912735 1 tasks_processing.go:71] worker 11 working on feature_gates task. I0319 19:20:03.912819 1 tasks_processing.go:71] worker 3 working on qemu_kubevirt_launcher_logs task. I0319 19:20:03.912953 1 tasks_processing.go:71] worker 24 working on ceph_cluster task. I0319 19:20:03.912460 1 tasks_processing.go:69] worker 31 listening for tasks. I0319 19:20:03.913146 1 tasks_processing.go:71] worker 31 working on nodenetworkstates task. I0319 19:20:03.913267 1 tasks_processing.go:71] worker 16 working on pod_network_connectivity_checks task. I0319 19:20:03.913315 1 tasks_processing.go:71] worker 5 working on dvo_metrics task. I0319 19:20:03.913349 1 tasks_processing.go:71] worker 26 working on install_plans task. I0319 19:20:03.913392 1 tasks_processing.go:71] worker 13 working on authentication task. I0319 19:20:03.913306 1 tasks_processing.go:71] worker 4 working on infrastructures task. I0319 19:20:03.913525 1 tasks_processing.go:71] worker 19 working on nodenetworkconfigurationpolicies task. I0319 19:20:03.913350 1 tasks_processing.go:71] worker 18 working on container_runtime_configs task. I0319 19:20:03.913987 1 tasks_processing.go:69] worker 8 listening for tasks. I0319 19:20:03.914049 1 tasks_processing.go:71] worker 8 working on clusterroles task. I0319 19:20:03.913343 1 tasks_processing.go:71] worker 20 working on openshift_logging task. I0319 19:20:03.913355 1 tasks_processing.go:71] worker 22 working on aggregated_monitoring_cr_names task. I0319 19:20:03.914682 1 tasks_processing.go:71] worker 28 working on ingress_certificates task. I0319 19:20:03.914913 1 tasks_processing.go:71] worker 21 working on metrics task. W0319 19:20:03.914988 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 19:20:03.915024 1 tasks_processing.go:71] worker 21 working on validating_webhook_configurations task. I0319 19:20:03.915107 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 50.199µs to process 0 records I0319 19:20:03.913341 1 tasks_processing.go:71] worker 25 working on config_maps task. I0319 19:20:03.915217 1 tasks_processing.go:71] worker 7 working on openshift_machine_api_events task. I0319 19:20:03.915746 1 tasks_processing.go:71] worker 29 working on oauths task. I0319 19:20:03.913360 1 tasks_processing.go:71] worker 30 working on machine_config_pools task. I0319 19:20:03.915980 1 tasks_processing.go:71] worker 6 working on mutating_webhook_configurations task. I0319 19:20:03.916072 1 tasks_processing.go:71] worker 27 working on support_secret task. I0319 19:20:03.913364 1 tasks_processing.go:71] worker 0 working on certificate_signing_requests task. I0319 19:20:03.913370 1 tasks_processing.go:71] worker 12 working on schedulers task. I0319 19:20:03.913373 1 tasks_processing.go:71] worker 14 working on nodes task. I0319 19:20:03.913385 1 tasks_processing.go:69] worker 15 listening for tasks. I0319 19:20:03.913312 1 tasks_processing.go:71] worker 17 working on sap_datahubs task. E0319 19:20:03.917317 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0319 19:20:03.917361 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 3.982014ms to process 0 records I0319 19:20:03.917394 1 tasks_processing.go:71] worker 16 working on ingress task. I0319 19:20:03.920074 1 tasks_processing.go:71] worker 24 working on jaegers task. I0319 19:20:03.920150 1 tasks_processing.go:71] worker 15 working on sap_config task. I0319 19:20:03.920378 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 7.061408ms to process 0 records E0319 19:20:03.920432 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0319 19:20:03.920473 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 7.716192ms to process 0 records I0319 19:20:03.920500 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 7.75036ms to process 0 records I0319 19:20:03.920524 1 tasks_processing.go:71] worker 23 working on service_accounts task. E0319 19:20:03.920531 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0319 19:20:03.920595 1 gather.go:177] gatherer "clusterconfig" function "machines" took 7.808042ms to process 0 records I0319 19:20:03.920614 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 7.395058ms to process 0 records I0319 19:20:03.920622 1 tasks_processing.go:71] worker 31 working on crds task. I0319 19:20:03.920513 1 tasks_processing.go:71] worker 10 working on proxies task. I0319 19:20:03.920774 1 tasks_processing.go:71] worker 20 working on operators_pods_and_events task. I0319 19:20:03.920800 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 6.594796ms to process 0 records I0319 19:20:03.920542 1 tasks_processing.go:71] worker 9 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0319 19:20:03.921307 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0319 19:20:03.921329 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0319 19:20:03.921336 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0319 19:20:03.921341 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0319 19:20:03.921359 1 controller.go:531] The operator is still being initialized I0319 19:20:03.921375 1 controller.go:554] The operator is healthy I0319 19:20:03.921424 1 tasks_processing.go:71] worker 2 working on pdbs task. I0319 19:20:03.921710 1 recorder.go:75] Recording config/network with fingerprint=407ebec2b3ce8d6d97af879e081686840e9046eac46eef31f0ef889349df0c14 I0319 19:20:03.921730 1 gather.go:177] gatherer "clusterconfig" function "networks" took 8.855045ms to process 1 records I0319 19:20:03.922098 1 tasks_processing.go:71] worker 11 working on operators task. I0319 19:20:03.922169 1 gather_logs.go:145] no pods in namespace were found I0319 19:20:03.922269 1 recorder.go:75] Recording config/featuregate with fingerprint=db02d0fec8e7ec37974fc3ef321d84d6090e5b959274c4ab4f4aa5bc6d1e2d2e I0319 19:20:03.922285 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 9.346748ms to process 1 records I0319 19:20:03.922292 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 9.30433ms to process 0 records I0319 19:20:03.922298 1 tasks_processing.go:71] worker 3 working on node_features task. I0319 19:20:03.922497 1 tasks_processing.go:71] worker 18 working on openstack_dataplanedeployments task. I0319 19:20:03.922511 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 8.785779ms to process 0 records I0319 19:20:03.922535 1 tasks_processing.go:71] worker 1 working on cluster_apiserver task. I0319 19:20:03.922610 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 10.064919ms to process 0 records I0319 19:20:03.923179 1 tasks_processing.go:71] worker 19 working on active_alerts task. I0319 19:20:03.923192 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 9.636872ms to process 0 records W0319 19:20:03.923207 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 19:20:03.923222 1 tasks_processing.go:71] worker 19 working on image_pruners task. I0319 19:20:03.923230 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 28.051µs to process 0 records I0319 19:20:03.923685 1 tasks_processing.go:71] worker 4 working on machine_sets task. I0319 19:20:03.924203 1 recorder.go:75] Recording config/infrastructure with fingerprint=beb8ebbf414f5349364a691dc9a56dc3a9e680bbca46ac98d883a71731ec7885 I0319 19:20:03.924217 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 10.202966ms to process 1 records I0319 19:20:03.927458 1 tasks_processing.go:71] worker 13 working on cost_management_metrics_configs task. I0319 19:20:03.927724 1 recorder.go:75] Recording config/authentication with fingerprint=8de47b0366ba6a1f8bf263980f6ac8307222d456fd15e54e4042f52febf46463 I0319 19:20:03.927743 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 14.008049ms to process 1 records E0319 19:20:03.930043 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0319 19:20:03.930071 1 tasks_processing.go:71] worker 7 working on tsdb_status task. I0319 19:20:03.930076 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 14.817886ms to process 0 records I0319 19:20:03.930179 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 7.569602ms to process 0 records I0319 19:20:03.930191 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 6.412127ms to process 0 records I0319 19:20:03.930198 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 7.81046ms to process 0 records W0319 19:20:03.930159 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 19:20:03.930236 1 tasks_processing.go:71] worker 18 working on image_registries task. I0319 19:20:03.930267 1 tasks_processing.go:71] worker 3 working on sap_pods task. I0319 19:20:03.930445 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=e2b158e7b04b3dcc535e64094b380e1fc86bf80c4397fee604612e53fd83a7aa I0319 19:20:03.930494 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=0b094d650bf15e0b57c2e6fa9399c433e7f2f210fd85d203a086da2cfa15d675 I0319 19:20:03.930540 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=e037817bbaf8bb44ddc417026aaff0860e1f15e9831332f54b7d254452fc4003 I0319 19:20:03.930550 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 14.141647ms to process 3 records I0319 19:20:03.930558 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 111.659µs to process 0 records I0319 19:20:03.930565 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 10.171608ms to process 0 records I0319 19:20:03.930572 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 12.992605ms to process 0 records I0319 19:20:03.930600 1 tasks_processing.go:71] worker 6 working on monitoring_persistent_volumes task. I0319 19:20:03.930705 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=d2fe5b634be2adcb263cb0f445672d9f6ee5d12bfd5d83cf288192b3cc12dba8 I0319 19:20:03.930243 1 tasks_processing.go:71] worker 4 working on container_images task. I0319 19:20:03.930893 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=3e54761cd516c26f2653bc1f6a15cceb803cef8e3f18fed225985789ea80e4b4 I0319 19:20:03.931149 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=17b124b359f085c3c6e4d738979a5d9a5a1c5395f22ba0783f2ff9f1814a9d45 I0319 19:20:03.931245 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=f63083ecbf6b9d1e9e39c14442ae66503acacd9ae935e3106627abbd25a1051b I0319 19:20:03.931304 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=1fde8fb92b33de2a284134aea9ed8f4ddd5a7108764f881a61facd487a8520b1 I0319 19:20:03.931377 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=6bf4fdb47f29709cc8095806ff738b4027b0e2738fb8fea693b9ccf75775f0c9 I0319 19:20:03.931435 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=4353e314dd0fb610c3971d95cfff7db3044790bb0f253de58920483ffe323aea I0319 19:20:03.931535 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=a0173a6719db28ea295bdac92f20e6b54e4808a386326ddd0141e5a739362277 I0319 19:20:03.931594 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=bda0c6b7ee9500f5a7c2aa424f570dbc2f0c884df469aefc4558b01c61402767 I0319 19:20:03.931646 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=e8af15c89726908a422022cbb5a085e585217a9aa4f41375ab5eeec5a1013cfe I0319 19:20:03.931697 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=16094a995e35ffa25a254e3a65df65953321668a2c6f93fc0128fa29dd3b46af I0319 19:20:03.931725 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 15.357898ms to process 11 records I0319 19:20:03.931752 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 10.252553ms to process 0 records I0319 19:20:03.931776 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 15.246794ms to process 0 records I0319 19:20:03.930900 1 tasks_processing.go:71] worker 21 working on image task. I0319 19:20:03.930906 1 tasks_processing.go:71] worker 7 working on openstack_dataplanenodesets task. I0319 19:20:03.930911 1 tasks_processing.go:71] worker 24 working on machine_autoscalers task. I0319 19:20:03.932248 1 tasks_processing.go:71] worker 0 working on machine_configs task. I0319 19:20:03.930915 1 tasks_processing.go:71] worker 17 working on openstack_controlplanes task. I0319 19:20:03.932487 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 15.808768ms to process 0 records I0319 19:20:03.931802 1 tasks_processing.go:71] worker 30 working on storage_classes task. I0319 19:20:03.931808 1 tasks_processing.go:71] worker 15 working on overlapping_namespace_uids task. I0319 19:20:03.932957 1 tasks_processing.go:71] worker 10 working on version task. I0319 19:20:03.933405 1 recorder.go:75] Recording config/proxy with fingerprint=7f8ef3e035caebe5bfd35ba62bcb8666bdd57e44eb062f4eb547eb468f9a8afe I0319 19:20:03.933453 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 12.280699ms to process 1 records I0319 19:20:03.937467 1 tasks_processing.go:71] worker 13 working on olm_operators task. I0319 19:20:03.937479 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 9.989091ms to process 0 records W0319 19:20:03.938156 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0319 19:20:03.939974 1 tasks_processing.go:71] worker 19 working on lokistack task. I0319 19:20:03.940396 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=e3849ee83132562ca423ee845b85a18f001bae52fcb3fe8c3d48a7bc593e7998 I0319 19:20:03.940416 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 16.733216ms to process 1 records E0319 19:20:03.940432 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0319 19:20:03.940814 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 24.147864ms to process 0 records I0319 19:20:03.940496 1 tasks_processing.go:71] worker 27 working on openstack_version task. I0319 19:20:03.941015 1 tasks_processing.go:71] worker 3 working on silenced_alerts task. W0319 19:20:03.941089 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 19:20:03.941241 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 10.020031ms to process 0 records I0319 19:20:03.941406 1 tasks_processing.go:74] worker 16 stopped. I0319 19:20:03.942025 1 recorder.go:75] Recording config/ingress with fingerprint=2cf622dc029a6605b0442207074d0d75df8ea17719f4d47fc00ad27f6ff6dfb0 I0319 19:20:03.942085 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 23.002438ms to process 1 records I0319 19:20:03.942398 1 recorder.go:75] Recording config/apiserver with fingerprint=6954b067cd77e9e5369b47ee67e66b72eefcca2a0fb5e9834831390e893f5f4e I0319 19:20:03.942451 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 18.622002ms to process 1 records I0319 19:20:03.942481 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 134.174µs to process 0 records I0319 19:20:03.942514 1 tasks_processing.go:74] worker 3 stopped. I0319 19:20:03.942538 1 tasks_processing.go:74] worker 1 stopped. I0319 19:20:03.942782 1 tasks_processing.go:74] worker 17 stopped. I0319 19:20:03.942824 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 10.343304ms to process 0 records I0319 19:20:03.943354 1 tasks_processing.go:74] worker 2 stopped. I0319 19:20:03.943586 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=d40bde1eaf4d7f9f73a7020b54cc898cfd59961a9a14009a809eaeedd03fc4fd I0319 19:20:03.943634 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=74201115092a8d37ff89d70f29001dbe43e8b30005f928bb9161c603c6c3c636 I0319 19:20:03.943663 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=e9db3c4e06f26b2b84966f2ba5d09b17bdb199ba8089a6ca7acb1bff46ca8c84 I0319 19:20:03.943679 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 21.878144ms to process 3 records I0319 19:20:03.943701 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 11.224189ms to process 0 records I0319 19:20:03.943734 1 tasks_processing.go:74] worker 24 stopped. I0319 19:20:03.943858 1 tasks_processing.go:74] worker 14 stopped. I0319 19:20:03.944763 1 recorder.go:75] Recording config/node/ip-10-0-0-164.ec2.internal with fingerprint=2aeb0393201f4d0391cd4ecad79ad82b1445fb5024c9cc927ae5b188296ce674 I0319 19:20:03.944826 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 26.41417ms to process 1 records I0319 19:20:03.944880 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 11.922902ms to process 0 records I0319 19:20:03.944950 1 tasks_processing.go:74] worker 7 stopped. I0319 19:20:03.944992 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=a848311935285e092aaa29f74a19c186d6c9f616a6ca987c429222d76f577b91 I0319 19:20:03.945033 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 27.357145ms to process 1 records I0319 19:20:03.944998 1 tasks_processing.go:74] worker 12 stopped. I0319 19:20:03.945352 1 recorder.go:75] Recording config/oauth with fingerprint=834236fb14d6d50a2484331b47396a803164c6380a94f53ed867d803c65ab879 I0319 19:20:03.945415 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 28.778328ms to process 1 records I0319 19:20:03.945359 1 tasks_processing.go:74] worker 29 stopped. I0319 19:20:03.945548 1 recorder.go:75] Recording config/image with fingerprint=382fe33adae19699ab8025a09d8f4c8b31ffebb5013522ed2be01ea40d3e027d I0319 19:20:03.945579 1 gather.go:177] gatherer "clusterconfig" function "image" took 12.822343ms to process 1 records I0319 19:20:03.945589 1 tasks_processing.go:74] worker 21 stopped. I0319 19:20:03.945685 1 tasks_processing.go:74] worker 27 stopped. I0319 19:20:03.945702 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 4.763453ms to process 0 records I0319 19:20:03.945915 1 tasks_processing.go:74] worker 8 stopped. I0319 19:20:03.946092 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=5b8ba44c281bf3ca5add9bd5a7bff57bfac8393a0dda93bcafbce95fe002b810 I0319 19:20:03.946204 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=b0c0ca9b9d8c365d91dcf36c81f578aa6d6aaa8015d63ebcb958dcde872c2889 I0319 19:20:03.946218 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 31.82923ms to process 2 records E0319 19:20:03.946239 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0319 19:20:03.946265 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0319 19:20:03.946279 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 13.658212ms to process 1 records I0319 19:20:03.946289 1 tasks_processing.go:74] worker 0 stopped. I0319 19:20:03.946732 1 tasks_processing.go:74] worker 19 stopped. I0319 19:20:03.946748 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 6.741378ms to process 0 records I0319 19:20:03.946788 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0319 19:20:03.946959 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0319 19:20:03.947091 1 operator.go:328] started I0319 19:20:03.947124 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0319 19:20:03.947802 1 tasks_processing.go:74] worker 6 stopped. I0319 19:20:03.947840 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 17.172379ms to process 0 records I0319 19:20:03.947966 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=1b9d43196ccfb1b740789e0e52b88c0dc7a05620f4dddcd1d5a690b9f5b4322f I0319 19:20:03.947984 1 tasks_processing.go:74] worker 30 stopped. I0319 19:20:03.947991 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=3c184f66006e935e69e2d312a71aa3a66b8e30565384ff6cfd0ed1bcc379fb83 I0319 19:20:03.948002 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 15.247648ms to process 2 records I0319 19:20:03.948695 1 tasks_processing.go:74] worker 18 stopped. I0319 19:20:03.949211 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=ef18b737f382eb98402caf459148ce8641302af8c5327ec7f371a8e65101c433 I0319 19:20:03.949234 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 18.430466ms to process 1 records I0319 19:20:03.949697 1 tasks_processing.go:74] worker 31 stopped. I0319 19:20:03.952402 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=5e178f709ab006e36eaabe5ceb33aeae86553f595289fb32dd8dfdfed15ce1cd I0319 19:20:03.952610 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=fce07fbc11efc288668f89a1e6bbf16948ea34a4a51087b42cc03f041524fcde I0319 19:20:03.952620 1 gather.go:177] gatherer "clusterconfig" function "crds" took 29.06181ms to process 2 records I0319 19:20:03.955359 1 tasks_processing.go:74] worker 9 stopped. I0319 19:20:03.955371 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 34.386752ms to process 0 records I0319 19:20:03.958462 1 tasks_processing.go:74] worker 13 stopped. I0319 19:20:03.958474 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 20.919035ms to process 0 records I0319 19:20:03.959079 1 tasks_processing.go:74] worker 22 stopped. I0319 19:20:03.959097 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 44.486985ms to process 0 records I0319 19:20:03.960578 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0319 19:20:03.960618 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0319 19:20:03.960690 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0319 19:20:03.964217 1 base_controller.go:82] Caches are synced for ConfigController I0319 19:20:03.964229 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0319 19:20:03.966930 1 prometheus_rules.go:88] Prometheus rules successfully created I0319 19:20:03.968141 1 controller.go:254] Source scaController *sca.Controller is not ready I0319 19:20:03.968151 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0319 19:20:03.968154 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0319 19:20:03.968158 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0319 19:20:03.968160 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0319 19:20:03.968178 1 controller.go:531] The operator is still being initialized I0319 19:20:03.968183 1 controller.go:554] The operator is healthy E0319 19:20:03.976754 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%271f2fb805-2db0-44d9-9889-e716c2a03c9b%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.6:55667->172.30.0.10:53: read: connection refused I0319 19:20:03.976767 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%271f2fb805-2db0-44d9-9889-e716c2a03c9b%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.6:55667->172.30.0.10:53: read: connection refused I0319 19:20:03.987770 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 19:20:04.005627 1 tasks_processing.go:74] worker 25 stopped. E0319 19:20:04.005642 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0319 19:20:04.005649 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0319 19:20:04.005655 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0319 19:20:04.005668 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=c8a9805405aae7d04b7f03c09c51097942070877009b6daebd5573ef8127446e I0319 19:20:04.005705 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0319 19:20:04.005713 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=3b70b9787487b548204d806ebebf90b0c2d460145e34cb33c0d8d3a7a011d907 I0319 19:20:04.005717 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0319 19:20:04.005721 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0319 19:20:04.005766 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0319 19:20:04.005775 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0319 19:20:04.005780 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 90.417918ms to process 7 records I0319 19:20:04.023917 1 tasks_processing.go:74] worker 10 stopped. I0319 19:20:04.024156 1 recorder.go:75] Recording config/version with fingerprint=7ebdf1f4a57dd7aa671a270cf6c6a322805aeb6ac89076122236a97685e6ef30 I0319 19:20:04.024170 1 recorder.go:75] Recording config/id with fingerprint=c06d803f3e4dd2fd0fd193d73f6942d24321da0ebaa00c3b11b276a91801e35f I0319 19:20:04.024177 1 gather.go:177] gatherer "clusterconfig" function "version" took 90.934976ms to process 2 records I0319 19:20:04.033099 1 tasks_processing.go:74] worker 4 stopped. I0319 19:20:04.033219 1 recorder.go:75] Recording config/running_containers with fingerprint=471c6c64aa9eb310da293dbf7c1cbb2010cdf6b16f0ed3530633913f4091a797 I0319 19:20:04.033240 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 102.148345ms to process 1 records I0319 19:20:04.034292 1 tasks_processing.go:74] worker 15 stopped. I0319 19:20:04.034327 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0319 19:20:04.034341 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 101.516472ms to process 1 records I0319 19:20:04.038135 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0319 19:20:04.041773 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.6:32963->172.30.0.10:53: read: connection refused I0319 19:20:04.041788 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.6:32963->172.30.0.10:53: read: connection refused I0319 19:20:04.047554 1 base_controller.go:82] Caches are synced for LoggingSyncer I0319 19:20:04.047568 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0319 19:20:04.056378 1 tasks_processing.go:74] worker 28 stopped. E0319 19:20:04.056391 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0319 19:20:04.056397 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2p4vh8val8ukmk9irdtni0ahl21inpgt-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2p4vh8val8ukmk9irdtni0ahl21inpgt-primary-cert-bundle-secret" not found I0319 19:20:04.056447 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=ecf71cb20d8db61e37f22b29f2afcc161e845dcd9171be450c79f37933a521df I0319 19:20:04.056459 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 141.640399ms to process 1 records I0319 19:20:04.353773 1 gather_cluster_operator_pods_and_events.go:121] Found 10 pods with 11 containers I0319 19:20:04.353786 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 2287802 bytes I0319 19:20:04.354732 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-zk58r pod in namespace openshift-dns (previous: false). I0319 19:20:04.605750 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-zk58r pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-zk58r\" is waiting to start: ContainerCreating" I0319 19:20:04.605768 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-zk58r\" is waiting to start: ContainerCreating" I0319 19:20:04.605777 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-zk58r pod in namespace openshift-dns (previous: false). I0319 19:20:04.762463 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-zk58r pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-zk58r\" is waiting to start: ContainerCreating" I0319 19:20:04.762481 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-zk58r\" is waiting to start: ContainerCreating" I0319 19:20:04.762490 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-7phts pod in namespace openshift-dns (previous: false). W0319 19:20:04.936137 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0319 19:20:04.959509 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 19:20:04.959527 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6f454d567d-4vncr pod in namespace openshift-image-registry (previous: false). I0319 19:20:05.158966 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-6f454d567d-4vncr pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-6f454d567d-4vncr\" is waiting to start: ContainerCreating" I0319 19:20:05.158983 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-6f454d567d-4vncr\" is waiting to start: ContainerCreating" I0319 19:20:05.158994 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-75897f5585-54rmw pod in namespace openshift-image-registry (previous: false). I0319 19:20:05.356149 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 19:20:05.356167 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-75897f5585-ng5cz pod in namespace openshift-image-registry (previous: false). I0319 19:20:05.369495 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0319 19:20:05.553705 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 19:20:05.553722 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-lxsq6 pod in namespace openshift-image-registry (previous: false). I0319 19:20:05.759488 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 19:20:05.759511 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5875786df-7wjmc pod in namespace openshift-ingress (previous: false). W0319 19:20:05.936667 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0319 19:20:05.953325 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0319 19:20:05.953348 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5875786df-8txll pod in namespace openshift-ingress (previous: false). I0319 19:20:05.972461 1 tasks_processing.go:74] worker 11 stopped. I0319 19:20:05.972522 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=4c7e7b24353984bf978e63e042305853bebdad4d8d0e6b424ad019460aef0189 I0319 19:20:05.972577 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=d2d1cd48bcf8a6989e1dbd709e628ab2b5187af1f7905ebd481bd21a3e4c65f4 I0319 19:20:05.972617 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0319 19:20:05.972694 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=4769825343f4c9c95a5d0d44a3f27c00928264bcf3a87f54568031d43a0944a1 I0319 19:20:05.972725 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0319 19:20:05.972750 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=58de25e93318d19d77ef1a7470e68ea295982888b6b983ccfac204107ac3baed I0319 19:20:05.972803 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=ff16a1040b6fdea48e1ca5b6933f00f02dbcba7460cf9f088d204788ff9aaa64 I0319 19:20:05.972900 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=1cd34094ad215a7902f64ab1c69b366c8c26a6fea3e4ee980b9802cbd334fc33 I0319 19:20:05.972945 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=2aa7ae80809c92456199bfab93b0b2079f6d689e20b4ef50d60ae65447df8f43 I0319 19:20:05.972966 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=67f38328649730cabdb83ee7871a57e94d5ceb0dce285da6840ff16d6e1c19b6 I0319 19:20:05.972976 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0319 19:20:05.972994 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=f7df9081245a141486ceedba0f7f71edf3bb9073fa085c5329e7e7a9bba8d5b5 I0319 19:20:05.973012 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0319 19:20:05.973034 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=2e2e66e7430d76c17c0c1ea7c94f578223a91d36a3504c82b1712e12c0692800 I0319 19:20:05.973049 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0319 19:20:05.973065 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=96bcc122c78f0552192c38fbbedb6e182a1dc0ff4b729cc585e35a69b1f4ad2a I0319 19:20:05.973075 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0319 19:20:05.973091 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=d97ec8112380818491ef3a8fd96998f51f1a6350fa08b8e2ed4af8b8861a9ffe I0319 19:20:05.973237 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=6b004c495ec9c397b5312bfa4af83795a03250ef31ec51b37eb09309dc99be3d I0319 19:20:05.973248 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0319 19:20:05.973256 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0319 19:20:05.973276 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0319 19:20:05.973306 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=49c237e30332e7025b8ff9ee2c27d6259049212e37f6918cf4363732277dbf57 I0319 19:20:05.973331 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=90537d3b7233f72bf4343cc6b90b7fd77afe414ec54869ec203edde8fb7b7110 I0319 19:20:05.973341 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0319 19:20:05.973368 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=df53e590ec6606776b6b64fb1a396669fd7bdf49ef7faf46dac6b0abab5e68dd I0319 19:20:05.973384 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0319 19:20:05.973403 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=8b18e7f6ac54f323b7f054b510bf7a92cab9ae60f8901f7433495048b2783fa8 I0319 19:20:05.973426 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=8fba5a8b4ea73a2e502ba19db7ce21423ee11dffc734452a456b72b28d6a6a16 I0319 19:20:05.973452 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=8d70527ef31563f42ed69192bc450192231df43f2a6e254b200bede16168aa44 I0319 19:20:05.973479 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=fb219a5c413617294addbce98f6946c533a7cb23289f0d2d6a25149fa1bf06d4 I0319 19:20:05.973512 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=ba5f3228906159bc552a481e99130796e0c541a5b8383a2568605e93cbb28751 I0319 19:20:05.973527 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0319 19:20:05.973586 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=9cbd183807ff67da115b0609df94c46b5f35582e4e9cc30f16691e759bd947cc I0319 19:20:05.973614 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0319 19:20:05.973627 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0319 19:20:05.973637 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.050334337s to process 36 records I0319 19:20:06.160035 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5875786df-8txll pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5875786df-8txll\" is waiting to start: ContainerCreating" I0319 19:20:06.160052 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5875786df-8txll\" is waiting to start: ContainerCreating" I0319 19:20:06.160064 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-69994fdfd9-xnngc pod in namespace openshift-ingress (previous: false). I0319 19:20:06.360064 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-69994fdfd9-xnngc pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-69994fdfd9-xnngc\" is waiting to start: ContainerCreating" I0319 19:20:06.360084 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-69994fdfd9-xnngc\" is waiting to start: ContainerCreating" I0319 19:20:06.360096 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-4p5wl pod in namespace openshift-ingress-canary (previous: false). I0319 19:20:06.559571 1 tasks_processing.go:74] worker 20 stopped. I0319 19:20:06.559655 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=ffc0ab509c9b997db02baf64171308431a7bd0058490e2eb7aa4fbc32c4c3304 I0319 19:20:06.559683 1 recorder.go:75] Recording events/openshift-dns with fingerprint=396fd5ba8252d4dbfabbb82c0630c0a38d66cec39912bdeb608c2b1e5ce3d69c I0319 19:20:06.559749 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=5f6864e653f3806bbf77d2aebb91163a0353ba05aafc494bf5f3fe2a8974176d I0319 19:20:06.559779 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=b01f2f400ea973488b8577f9b754fc0f365d6ee028039cd7888558b258dc6b0b I0319 19:20:06.559821 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=8839a751a2eb1992b5b514d52f5863e4302aea00864d43361fbee6918b12d366 I0319 19:20:06.559842 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=f5efca77f80200afcb85701e277c9a956b77f1b2d96cde27decdb91d67998712 I0319 19:20:06.559849 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-4p5wl/serve-healthcheck-canary_current.log with fingerprint=9069a2f29dd8bdc741e62cdda86bf8ab5e1cc504b78a0aafaa37eb37493be389 I0319 19:20:06.559856 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 2.638737853s to process 7 records W0319 19:20:06.935600 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0319 19:20:07.937979 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0319 19:20:08.935142 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0319 19:20:08.935165 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0319 19:20:08.935181 1 tasks_processing.go:74] worker 5 stopped. E0319 19:20:08.935191 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0319 19:20:08.935201 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0319 19:20:08.935215 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0319 19:20:08.935224 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.021822293s to process 1 records I0319 19:20:16.352344 1 tasks_processing.go:74] worker 26 stopped. I0319 19:20:16.352377 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0319 19:20:16.352389 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.438940132s to process 1 records I0319 19:20:16.934720 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 19:20:17.096142 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 19:20:17.129654 1 tasks_processing.go:74] worker 23 stopped. I0319 19:20:17.129944 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=7c1a3e3c431de69ce4ee6fb33aad2158a3d38446a91ec5899ca71ceaf725097e I0319 19:20:17.129966 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.209070022s to process 1 records E0319 19:20:17.130049 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.217s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0319 19:20:17.131151 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0319 19:20:17.131166 1 periodic.go:216] Running workloads gatherer I0319 19:20:17.131182 1 tasks_processing.go:45] number of workers: 2 I0319 19:20:17.131196 1 tasks_processing.go:69] worker 1 listening for tasks. I0319 19:20:17.131202 1 tasks_processing.go:71] worker 1 working on workload_info task. I0319 19:20:17.131205 1 tasks_processing.go:69] worker 0 listening for tasks. I0319 19:20:17.131276 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0319 19:20:17.160532 1 tasks_processing.go:74] worker 0 stopped. I0319 19:20:17.160547 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 29.239194ms to process 0 records I0319 19:20:17.201894 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 19s for image data I0319 19:20:17.211023 1 gather_workloads_info.go:387] No image sha256:5efa227ddac82e56eddc03761e220af1f2d6a4b22ee00b2baf426523f26c0319 (10ms) I0319 19:20:17.223042 1 gather_workloads_info.go:387] No image sha256:bbb5af28bbed69e8189825fdc6665f4e8dbcd0c0c83fa479943ea102fe9452d6 (12ms) I0319 19:20:17.233997 1 gather_workloads_info.go:387] No image sha256:60648d27017709a420e43353f9a83d7c466017b3b771200672a4dbc3c3f7a3ae (11ms) I0319 19:20:17.244189 1 gather_workloads_info.go:387] No image sha256:05376e7e2304d22dd866d8e25dd1e5ff8988b2232059fadfd12cb440af785635 (10ms) I0319 19:20:17.254997 1 gather_workloads_info.go:387] No image sha256:26e4f5b039a340bc1ae080fb9d92799700b57ae5df45120768f516682d6cc54b (11ms) I0319 19:20:17.267416 1 gather_workloads_info.go:387] No image sha256:41e0dcd2f91df9234249f168b3fbe6d55cfdb3ebd2e598d5edfa47f905c046cb (12ms) I0319 19:20:17.277341 1 gather_workloads_info.go:387] No image sha256:17e68da37c74bdc95a4eb43b51ce671027807ee610a4ee5237324a8c0dc9d044 (10ms) I0319 19:20:17.289700 1 gather_workloads_info.go:387] No image sha256:7dd9584d291967387f2a1ade991ad3bf8565bb59f49b013217951399e8273498 (12ms) I0319 19:20:17.300714 1 gather_workloads_info.go:387] No image sha256:c85dd52f4140690eef0d762834ab8b9af2faedbbd3bd9dd5616d4cd0bebd4740 (11ms) I0319 19:20:17.302172 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 19:20:17.312647 1 gather_workloads_info.go:387] No image sha256:77d8ddf2957025a39616328b5cf2b012de6cedce63fb79eebc5270d7b3401574 (12ms) I0319 19:20:17.323140 1 gather_workloads_info.go:387] No image sha256:862d5befeba504d5058264cf65f16fc307eea4eb3914fbbbf40fbe61714b0958 (10ms) I0319 19:20:17.418690 1 gather_workloads_info.go:387] No image sha256:11a864539ae77a94afcf8d2e1abf9168081c5081a1bc06088f4be5206cdc02df (96ms) I0319 19:20:17.514667 1 gather_workloads_info.go:387] No image sha256:3e262d5207d5e2077e4a14f4fbddc70d4d2651dbaaf955147013feb58a18cddd (96ms) I0319 19:20:17.615152 1 gather_workloads_info.go:387] No image sha256:db40a3d4e50016c645fdb4a1f389dfcdbc9febc10c174816d681abe5c14153cd (100ms) I0319 19:20:17.712257 1 gather_workloads_info.go:387] No image sha256:0b2cc6aadb1abc1d35a55da0c9884eb9426d7ff7a97174daeb09bb1e1a23cfae (97ms) I0319 19:20:17.811183 1 gather_workloads_info.go:387] No image sha256:3e889ea85c9c4748c548b84ae135a8adc88f42026ce8073538ca2fe7d89b8004 (99ms) I0319 19:20:17.912168 1 gather_workloads_info.go:387] No image sha256:ab32e70fb149ae7470c941ca73bcfe9eba540bd5058a3d697a3ddf96597d0132 (101ms) I0319 19:20:18.011260 1 gather_workloads_info.go:387] No image sha256:0c90ca6acbe2f4fdbd253ae22f5b3286247c6940321f7d2b9801aea21ef5269f (99ms) I0319 19:20:18.111754 1 gather_workloads_info.go:387] No image sha256:04ee35aa6f87e954d386228808017cf2938c5a96a052059c6767078d5f3ad8d6 (100ms) I0319 19:20:18.212073 1 gather_workloads_info.go:387] No image sha256:50b7c5185c4a6e63145753415c92723c59136d14b951ea0f4f06739b00893329 (100ms) I0319 19:20:18.312430 1 gather_workloads_info.go:387] No image sha256:04c7a2673571e9069c934d86b4308c0f57b2715b45db500837b7708ae1f20ae7 (100ms) I0319 19:20:18.412160 1 gather_workloads_info.go:387] No image sha256:188e4e2d697e2ca3c7c16772c0ab12253f00bd8433d8691ed7058fe7bafbbbf4 (100ms) I0319 19:20:18.515998 1 gather_workloads_info.go:387] No image sha256:dd6a017092d57ab3a6663f3822a0fa0e3bce4365859d9522dd173405276cd49a (104ms) I0319 19:20:18.622715 1 gather_workloads_info.go:387] No image sha256:1c325fa866a794c56a7b30ca9d4fe225606566651bb12a2ecce8f60cbddb2809 (107ms) I0319 19:20:18.712008 1 gather_workloads_info.go:387] No image sha256:2a4f6a54cd5135c8121832236063419071fa5d00c10a051c197d3d01370e4d9c (89ms) I0319 19:20:18.812200 1 gather_workloads_info.go:387] No image sha256:7ea1ff613255c1ee67b34ec82b68ec6f5a4a6d283620a8e0ad142d4a31eb9211 (100ms) I0319 19:20:18.911598 1 gather_workloads_info.go:387] No image sha256:25d9148ea34149935323231c64952679727b6f7f235febc3be1df63e0bf18c5e (99ms) I0319 19:20:19.012120 1 gather_workloads_info.go:387] No image sha256:1bf773bc4ec02bf6d68c82f32175366b01fb24f6da8c724b7621748692aec7dc (101ms) I0319 19:20:19.110991 1 gather_workloads_info.go:387] No image sha256:07f14370b57046dc2ac49be933bb2248aaf5943e8fb56ce9703eaa7c673b999f (99ms) I0319 19:20:19.211895 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (101ms) I0319 19:20:19.312112 1 gather_workloads_info.go:387] No image sha256:ff8a8892f37210c582b3e526cbd2652b02ef9007d57be12b404434374dd0116a (100ms) I0319 19:20:19.411295 1 gather_workloads_info.go:387] No image sha256:79fca57a4fd5ef00472a287218b84061b941905c61fcc55e89073dd93fa77648 (99ms) I0319 19:20:19.411323 1 tasks_processing.go:74] worker 1 stopped. E0319 19:20:19.411332 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0319 19:20:19.411556 1 recorder.go:75] Recording config/workload_info with fingerprint=48bfc79422303b02e30c2668925ee7e8c6c109ce3edf810479ed603e02c35078 I0319 19:20:19.411570 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.2801134s to process 1 records E0319 19:20:19.411592 1 periodic.go:254] "Unhandled Error" err="workloads failed after 2.28s with: function \"workload_info\" failed with an error" I0319 19:20:19.412690 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0319 19:20:19.412703 1 periodic.go:216] Running conditional gatherer I0319 19:20:19.418495 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.5/gathering_rules I0319 19:20:19.425773 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.21.5/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.6:53910->172.30.0.10:53: read: connection refused E0319 19:20:19.426029 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0319 19:20:19.426085 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0319 19:20:19.432801 1 conditional_gatherer.go:392] cluster version is '4.21.5' E0319 19:20:19.432813 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 19:20:19.432818 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 19:20:19.432828 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 19:20:19.432832 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 19:20:19.432835 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 19:20:19.432838 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 19:20:19.432841 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 19:20:19.432843 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0319 19:20:19.432846 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0319 19:20:19.432893 1 tasks_processing.go:45] number of workers: 3 I0319 19:20:19.432908 1 tasks_processing.go:69] worker 2 listening for tasks. I0319 19:20:19.432913 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0319 19:20:19.432919 1 tasks_processing.go:69] worker 0 listening for tasks. I0319 19:20:19.432932 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0319 19:20:19.432932 1 tasks_processing.go:69] worker 1 listening for tasks. I0319 19:20:19.432941 1 tasks_processing.go:71] worker 1 working on conditional_gatherer_rules task. I0319 19:20:19.432950 1 tasks_processing.go:74] worker 0 stopped. I0319 19:20:19.432996 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0319 19:20:19.433010 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.931µs to process 1 records I0319 19:20:19.433072 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0319 19:20:19.433082 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.292µs to process 1 records I0319 19:20:19.433087 1 tasks_processing.go:74] worker 1 stopped. I0319 19:20:19.433139 1 tasks_processing.go:74] worker 2 stopped. I0319 19:20:19.433150 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 221.187µs to process 0 records I0319 19:20:19.433168 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.21.5/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.6:53910->172.30.0.10:53: read: connection refused I0319 19:20:19.433185 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0319 19:20:19.454887 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=92d3dbcd5a97dfc932cae463455487a01cd3b46ba685946d66e5f9c38a06fac1 I0319 19:20:19.455004 1 diskrecorder.go:70] Writing 99 records to /var/lib/insights-operator/insights-2026-03-19-192019.tar.gz I0319 19:20:19.461067 1 diskrecorder.go:51] Wrote 99 records to disk in 6ms I0319 19:20:19.461092 1 periodic.go:285] Gathering cluster info every 2h0m0s I0319 19:20:19.461106 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0319 19:20:30.954101 1 configmapobserver.go:84] configmaps "insights-config" not found I0319 19:21:13.089017 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="3f108b8195088b60ed82df1e1b5bbfe72570523316fb3dc3cab92b0c6c4473cf") W0319 19:21:13.089055 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0319 19:21:13.089109 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="7c0387625cbe86bea6b420ac4fd8567bc0fe2fd5291d666e44a7a9bf157ec543")