W0314 15:15:35.214891 1 cmd.go:257] Using insecure, self-signed certificates I0314 15:15:35.933345 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0314 15:15:35.933662 1 observer_polling.go:159] Starting file observer I0314 15:15:36.891314 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0314 15:15:36.892410 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0314 15:15:36.892889 1 secure_serving.go:57] Forcing use of http/1.1 only W0314 15:15:36.892981 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0314 15:15:36.893012 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0314 15:15:36.893024 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0314 15:15:36.893035 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0314 15:15:36.893042 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0314 15:15:36.893047 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0314 15:15:36.893625 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0314 15:15:36.897419 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0314 15:15:36.897436 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0314 15:15:36.897425 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0314 15:15:36.897464 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0314 15:15:36.897433 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0314 15:15:36.897478 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0314 15:15:36.897636 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0314 15:15:36.897853 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"d7b940b4-6e08-4457-8253-efe4febc8a9c", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0314 15:15:36.897933 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-4229245323/tls.crt::/tmp/serving-cert-4229245323/tls.key" I0314 15:15:36.898229 1 secure_serving.go:213] Serving securely on [::]:8443 I0314 15:15:36.898262 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0314 15:15:36.902473 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0314 15:15:36.902503 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0314 15:15:36.902531 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0314 15:15:36.906930 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0314 15:15:36.906952 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0314 15:15:36.910973 1 secretconfigobserver.go:119] support secret does not exist I0314 15:15:36.914958 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0314 15:15:36.918412 1 secretconfigobserver.go:119] support secret does not exist I0314 15:15:36.923572 1 recorder.go:161] Pruning old reports every 7h59m14s, max age is 288h0m0s I0314 15:15:36.927984 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0314 15:15:36.927996 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0314 15:15:36.928008 1 insightsreport.go:296] Starting report retriever I0314 15:15:36.928005 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0314 15:15:36.928015 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0314 15:15:36.928033 1 periodic.go:209] Running clusterconfig gatherer I0314 15:15:36.928092 1 tasks_processing.go:45] number of workers: 32 I0314 15:15:36.928118 1 tasks_processing.go:69] worker 1 listening for tasks. I0314 15:15:36.928123 1 tasks_processing.go:69] worker 8 listening for tasks. I0314 15:15:36.928125 1 tasks_processing.go:71] worker 1 working on machine_configs task. I0314 15:15:36.928130 1 tasks_processing.go:69] worker 31 listening for tasks. I0314 15:15:36.928136 1 tasks_processing.go:69] worker 0 listening for tasks. I0314 15:15:36.928138 1 tasks_processing.go:71] worker 31 working on image task. I0314 15:15:36.928144 1 tasks_processing.go:69] worker 20 listening for tasks. I0314 15:15:36.928149 1 tasks_processing.go:69] worker 9 listening for tasks. I0314 15:15:36.928150 1 tasks_processing.go:69] worker 4 listening for tasks. I0314 15:15:36.928159 1 tasks_processing.go:69] worker 2 listening for tasks. I0314 15:15:36.928159 1 tasks_processing.go:69] worker 15 listening for tasks. I0314 15:15:36.928166 1 tasks_processing.go:69] worker 16 listening for tasks. I0314 15:15:36.928169 1 tasks_processing.go:69] worker 18 listening for tasks. I0314 15:15:36.928174 1 tasks_processing.go:69] worker 19 listening for tasks. I0314 15:15:36.928178 1 tasks_processing.go:69] worker 17 listening for tasks. I0314 15:15:36.928182 1 tasks_processing.go:69] worker 10 listening for tasks. I0314 15:15:36.928186 1 tasks_processing.go:69] worker 12 listening for tasks. I0314 15:15:36.928183 1 tasks_processing.go:69] worker 3 listening for tasks. I0314 15:15:36.928187 1 tasks_processing.go:69] worker 14 listening for tasks. I0314 15:15:36.928195 1 tasks_processing.go:69] worker 7 listening for tasks. I0314 15:15:36.928191 1 tasks_processing.go:69] worker 6 listening for tasks. I0314 15:15:36.928199 1 tasks_processing.go:69] worker 13 listening for tasks. I0314 15:15:36.928198 1 tasks_processing.go:69] worker 11 listening for tasks. I0314 15:15:36.928198 1 tasks_processing.go:69] worker 5 listening for tasks. I0314 15:15:36.928203 1 tasks_processing.go:69] worker 26 listening for tasks. I0314 15:15:36.928213 1 tasks_processing.go:69] worker 27 listening for tasks. I0314 15:15:36.928208 1 tasks_processing.go:69] worker 23 listening for tasks. I0314 15:15:36.928217 1 tasks_processing.go:69] worker 28 listening for tasks. I0314 15:15:36.928209 1 tasks_processing.go:69] worker 29 listening for tasks. I0314 15:15:36.928223 1 tasks_processing.go:69] worker 30 listening for tasks. I0314 15:15:36.928212 1 tasks_processing.go:69] worker 24 listening for tasks. I0314 15:15:36.928215 1 tasks_processing.go:69] worker 21 listening for tasks. I0314 15:15:36.928131 1 tasks_processing.go:71] worker 8 working on sap_datahubs task. I0314 15:15:36.928268 1 tasks_processing.go:71] worker 21 working on container_runtime_configs task. I0314 15:15:36.928219 1 tasks_processing.go:69] worker 25 listening for tasks. I0314 15:15:36.928298 1 tasks_processing.go:71] worker 25 working on schedulers task. I0314 15:15:36.928303 1 tasks_processing.go:71] worker 3 working on openstack_dataplanenodesets task. I0314 15:15:36.928222 1 tasks_processing.go:69] worker 22 listening for tasks. I0314 15:15:36.928357 1 tasks_processing.go:71] worker 22 working on config_maps task. I0314 15:15:36.928374 1 tasks_processing.go:71] worker 15 working on active_alerts task. W0314 15:15:36.928405 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0314 15:15:36.928421 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 29.03µs to process 0 records I0314 15:15:36.928430 1 tasks_processing.go:71] worker 0 working on cluster_apiserver task. I0314 15:15:36.928442 1 tasks_processing.go:71] worker 9 working on support_secret task. I0314 15:15:36.928483 1 tasks_processing.go:71] worker 14 working on clusterroles task. I0314 15:15:36.928504 1 tasks_processing.go:71] worker 16 working on storage_cluster task. I0314 15:15:36.928576 1 tasks_processing.go:71] worker 4 working on oauths task. I0314 15:15:36.928638 1 tasks_processing.go:71] worker 20 working on openstack_version task. I0314 15:15:36.928685 1 tasks_processing.go:71] worker 15 working on olm_operators task. I0314 15:15:36.928700 1 tasks_processing.go:71] worker 17 working on feature_gates task. I0314 15:15:36.928768 1 tasks_processing.go:71] worker 18 working on openshift_machine_api_events task. I0314 15:15:36.928802 1 tasks_processing.go:71] worker 7 working on validating_webhook_configurations task. I0314 15:15:36.928868 1 tasks_processing.go:71] worker 23 working on openstack_dataplanedeployments task. I0314 15:15:36.928781 1 tasks_processing.go:71] worker 19 working on proxies task. I0314 15:15:36.928816 1 tasks_processing.go:71] worker 12 working on pdbs task. I0314 15:15:36.928798 1 tasks_processing.go:71] worker 27 working on silenced_alerts task. W0314 15:15:36.929095 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0314 15:15:36.928791 1 tasks_processing.go:71] worker 10 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0314 15:15:36.928877 1 tasks_processing.go:71] worker 30 working on container_images task. I0314 15:15:36.928827 1 tasks_processing.go:71] worker 11 working on dvo_metrics task. I0314 15:15:36.928833 1 tasks_processing.go:71] worker 6 working on install_plans task. I0314 15:15:36.928836 1 tasks_processing.go:71] worker 13 working on storage_classes task. I0314 15:15:36.928842 1 tasks_processing.go:71] worker 29 working on sap_pods task. I0314 15:15:36.928883 1 tasks_processing.go:71] worker 24 working on operators task. I0314 15:15:36.928693 1 tasks_processing.go:71] worker 2 working on authentication task. I0314 15:15:36.928894 1 tasks_processing.go:71] worker 5 working on sap_config task. I0314 15:15:36.928901 1 tasks_processing.go:71] worker 26 working on machine_autoscalers task. I0314 15:15:36.928908 1 tasks_processing.go:71] worker 28 working on image_pruners task. I0314 15:15:36.929130 1 tasks_processing.go:71] worker 27 working on machines task. I0314 15:15:36.929142 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 50.365µs to process 0 records I0314 15:15:36.934954 1 tasks_processing.go:71] worker 9 working on nodenetworkstates task. E0314 15:15:36.934963 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0314 15:15:36.934974 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 6.497866ms to process 0 records I0314 15:15:36.935013 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0314 15:15:36.935033 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0314 15:15:36.935040 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0314 15:15:36.935044 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0314 15:15:36.935057 1 controller.go:489] The operator is still being initialized I0314 15:15:36.935065 1 controller.go:512] The operator is healthy I0314 15:15:36.935144 1 tasks_processing.go:71] worker 31 working on jaegers task. I0314 15:15:36.935260 1 recorder.go:75] Recording config/image with fingerprint=d707bec8060f5feb8cb40c67e9840bab0450172cce46bf4aec5c919c8572ad3b I0314 15:15:36.935276 1 gather.go:177] gatherer "clusterconfig" function "image" took 6.992465ms to process 1 records I0314 15:15:36.939128 1 tasks_processing.go:71] worker 20 working on nodenetworkconfigurationpolicies task. I0314 15:15:36.939156 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 10.466752ms to process 0 records I0314 15:15:36.939165 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 10.227272ms to process 0 records I0314 15:15:36.939170 1 tasks_processing.go:71] worker 23 working on certificate_signing_requests task. I0314 15:15:36.939183 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 10.899043ms to process 0 records I0314 15:15:36.939201 1 tasks_processing.go:71] worker 21 working on ingress task. I0314 15:15:36.941940 1 tasks_processing.go:71] worker 5 working on version task. I0314 15:15:36.941963 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 12.110778ms to process 0 records I0314 15:15:36.941977 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 12.523632ms to process 0 records I0314 15:15:36.941984 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 13.642101ms to process 0 records I0314 15:15:36.941992 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 12.103465ms to process 0 records I0314 15:15:36.941999 1 tasks_processing.go:71] worker 26 working on tsdb_status task. W0314 15:15:36.942022 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0314 15:15:36.942032 1 tasks_processing.go:71] worker 3 working on qemu_kubevirt_launcher_logs task. I0314 15:15:36.942041 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 28.79µs to process 0 records I0314 15:15:36.942034 1 tasks_processing.go:71] worker 26 working on service_accounts task. I0314 15:15:36.942046 1 tasks_processing.go:71] worker 29 working on ceph_cluster task. I0314 15:15:36.942074 1 tasks_processing.go:71] worker 27 working on ingress_certificates task. E0314 15:15:36.942113 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0314 15:15:36.942130 1 gather.go:177] gatherer "clusterconfig" function "machines" took 12.10236ms to process 0 records I0314 15:15:36.942208 1 tasks_processing.go:71] worker 15 working on machine_config_pools task. I0314 15:15:36.942224 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 13.506271ms to process 0 records I0314 15:15:36.952943 1 tasks_processing.go:71] worker 0 working on openstack_controlplanes task. I0314 15:15:36.953169 1 recorder.go:75] Recording config/apiserver with fingerprint=2234d4bbe53d0f776151440e0ab3afaada19211948a039b9734fd2dae196d984 I0314 15:15:36.953188 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 24.504468ms to process 1 records I0314 15:15:36.953221 1 tasks_processing.go:71] worker 8 working on cost_management_metrics_configs task. I0314 15:15:36.953268 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 24.98073ms to process 0 records I0314 15:15:36.953450 1 tasks_processing.go:71] worker 16 working on nodes task. I0314 15:15:36.953466 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 24.924112ms to process 0 records I0314 15:15:36.953909 1 tasks_processing.go:71] worker 2 working on pod_network_connectivity_checks task. I0314 15:15:36.956581 1 recorder.go:75] Recording config/authentication with fingerprint=1f742e41e7f744a883dcf6e5666b55ac4c9d58888d66ad84100d070c588344da I0314 15:15:36.956970 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 24.281162ms to process 1 records I0314 15:15:36.957690 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 19.201546ms to process 0 records I0314 15:15:36.957745 1 tasks_processing.go:71] worker 31 working on aggregated_monitoring_cr_names task. I0314 15:15:36.958357 1 tasks_processing.go:71] worker 17 working on overlapping_namespace_uids task. I0314 15:15:36.958538 1 recorder.go:75] Recording config/featuregate with fingerprint=011f5d2e561a962c5c41c519e907cb47a932c8ffe6d49e1aeab079890c4d9239 I0314 15:15:36.958652 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 26.078316ms to process 1 records I0314 15:15:36.958850 1 tasks_processing.go:71] worker 4 working on openshift_logging task. I0314 15:15:36.959809 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0314 15:15:36.959905 1 operator.go:288] started I0314 15:15:36.959813 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0314 15:15:36.959960 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0314 15:15:36.959985 1 recorder.go:75] Recording config/oauth with fingerprint=6cbfca3addf47e5747743d5cc329a9d303d57cd48cc1857f58525080447ef5a6 I0314 15:15:36.960064 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 26.324831ms to process 1 records I0314 15:15:36.960215 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=c25353fa6fc6ca8e1664080a1dcbb0d573f5ea244b91df9cd937b3c1158d7ab1 I0314 15:15:36.960261 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 26.755282ms to process 1 records I0314 15:15:36.960295 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 20.199622ms to process 0 records I0314 15:15:36.960301 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 16.091048ms to process 0 records I0314 15:15:36.960339 1 tasks_processing.go:71] worker 9 working on machine_sets task. I0314 15:15:36.960420 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=34296f826802396d53d5cbdeba06608f69590d51ba8cc971ef6a672c7462c3bc I0314 15:15:36.960490 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=b345da331bce611ea4a22cc63c41ea3292012c61395b7add3d95ed1cc2bd52cb I0314 15:15:36.960541 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 25.925786ms to process 2 records I0314 15:15:36.960572 1 tasks_processing.go:71] worker 20 working on metrics task. W0314 15:15:36.960632 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0314 15:15:36.960675 1 tasks_processing.go:71] worker 13 working on node_logs task. I0314 15:15:36.960837 1 recorder.go:75] Recording config/ingress with fingerprint=5f26ee2185e9120fec08a89b7d276487aac5473aa2a4c6af14e6f8d18fccf635 I0314 15:15:36.960921 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 16.578949ms to process 1 records I0314 15:15:36.961069 1 recorder.go:75] Recording config/proxy with fingerprint=d0a9ba32eb800cb2662667afc13dce37c447aa0d8a862eb85b72426e0b3c040e I0314 15:15:36.961112 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 26.905534ms to process 1 records I0314 15:15:36.961162 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 27.25074ms to process 0 records I0314 15:15:36.961315 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=02e1202560e10d1b67e261c2b03bd2edcdc78f288dfef68484dbd76877124e9d I0314 15:15:36.961375 1 tasks_processing.go:71] worker 12 working on networks task. I0314 15:15:36.960246 1 tasks_processing.go:71] worker 25 working on lokistack task. I0314 15:15:36.961395 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=c3aa555785e9969979a26aff6d504fee0316325cffe44cba533b41c90945c5f8 I0314 15:15:36.961743 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=4462a2d4dcc8463e4320eb9b3b0ebec5bc6918e741056d1ef8811a3e75ab5dee I0314 15:15:36.961763 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 27.068911ms to process 3 records I0314 15:15:36.961316 1 tasks_processing.go:71] worker 21 working on machine_healthchecks task. I0314 15:15:36.961983 1 tasks_processing.go:71] worker 28 working on infrastructures task. I0314 15:15:36.961344 1 tasks_processing.go:71] worker 19 working on monitoring_persistent_volumes task. I0314 15:15:36.962145 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=c834769a4ce3d67348a875ce8962ac150ab34fcc16b0cc09cc0fd3a473275acf I0314 15:15:36.962158 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 27.891144ms to process 1 records I0314 15:15:36.962419 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=52822d5fe227425e7b5a0bb645e7f247cc5a6411b5fa0aae9b45f384a1e50897 I0314 15:15:36.961353 1 tasks_processing.go:71] worker 18 working on crds task. I0314 15:15:36.962614 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=3337d7cc7c67af277354d3b8f79b1a1298b0ea74be1993e0d4bf7f32663fd1c7 I0314 15:15:36.962677 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=c1c706fc69eb580048b8ce7ec9b3f31535cc0605bb97156bf2627be7834e5d8b I0314 15:15:36.962765 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=1351945c546b6ee44fe82b5ae70f4f5fbac98f3b30fd6f8138afabb09d17b022 I0314 15:15:36.962777 1 tasks_processing.go:71] worker 7 working on operators_pods_and_events task. I0314 15:15:36.962810 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=e8763d089d1803782de172bb217108221e75257fca69b182225762aa2035559b I0314 15:15:36.962855 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=753c654666f29645b6c5999ea88e9446ca379b313280e34ba83021c90010cf10 I0314 15:15:36.962921 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=c0e8f08e7b46971eb084780eb3d9fe36a15516a5a5a9a2a4fc8d4a85433a1d94 I0314 15:15:36.962968 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=558027a3454746c28db185a61870f4198d63f9e1be966f0d11a62d149ae4ad68 I0314 15:15:36.963053 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=a9d61ded9890ffae226ed9f1ee0f67c5d30c4aa12bd959c5efdd5b320e03e46c I0314 15:15:36.963161 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=0cb0f2a4faf64a1c68e3d0c5a3105a2dd06ed379bc898b45ad2324c5facffd40 I0314 15:15:36.963200 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 29.325376ms to process 10 records I0314 15:15:36.963243 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 16.990429ms to process 0 records I0314 15:15:36.963281 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 53.852µs to process 0 records I0314 15:15:36.963311 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 22.188385ms to process 0 records I0314 15:15:36.963309 1 tasks_processing.go:71] worker 29 working on image_registries task. I0314 15:15:36.963389 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 9.664777ms to process 0 records E0314 15:15:36.963425 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0314 15:15:36.963454 1 tasks_processing.go:74] worker 2 stopped. I0314 15:15:36.963456 1 tasks_processing.go:74] worker 8 stopped. I0314 15:15:36.963328 1 tasks_processing.go:74] worker 23 stopped. I0314 15:15:36.963458 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 9.124554ms to process 0 records I0314 15:15:36.963522 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 10.1961ms to process 0 records I0314 15:15:36.963556 1 tasks_processing.go:74] worker 0 stopped. I0314 15:15:36.963319 1 tasks_processing.go:71] worker 20 working on mutating_webhook_configurations task. I0314 15:15:36.964324 1 tasks_processing.go:74] worker 16 stopped. I0314 15:15:36.964894 1 recorder.go:75] Recording config/node/ip-10-0-2-164.ec2.internal with fingerprint=25184ca99920e64bb4ae6d9a2ed68ea6ff9abdd548f7717d62591f528ee14149 I0314 15:15:36.964913 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 10.855331ms to process 1 records W0314 15:15:36.964982 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0314 15:15:36.965005 1 tasks_processing.go:74] worker 14 stopped. I0314 15:15:36.965092 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=bec5483bd38286678bce785807595f3917e04ede72fb4ddc44123d0617aacac2 I0314 15:15:36.965383 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=02663cdaa7fa2f6d31fe83ca5667d5102115edd53e2b7ef6e3242aeb3fda6d9d I0314 15:15:36.965432 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 35.96661ms to process 2 records I0314 15:15:36.967972 1 tasks_processing.go:74] worker 4 stopped. I0314 15:15:36.968034 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 9.102776ms to process 0 records I0314 15:15:36.968928 1 tasks_processing.go:74] worker 30 stopped. I0314 15:15:36.969074 1 recorder.go:75] Recording config/running_containers with fingerprint=493a58ce4ebdd1691926ef5efeeea7f0d4789fd2c6919b96dfa7d83426125d3b I0314 15:15:36.969096 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 39.67327ms to process 1 records I0314 15:15:36.974746 1 tasks_processing.go:74] worker 25 stopped. I0314 15:15:36.974765 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 13.205749ms to process 0 records I0314 15:15:36.974867 1 tasks_processing.go:74] worker 21 stopped. E0314 15:15:36.974880 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0314 15:15:36.974891 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 12.927706ms to process 0 records I0314 15:15:36.974919 1 tasks_processing.go:74] worker 9 stopped. I0314 15:15:36.974931 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 14.541322ms to process 0 records I0314 15:15:36.975068 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0314 15:15:36.975081 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0314 15:15:36.975087 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0314 15:15:36.975092 1 controller.go:212] Source scaController *sca.Controller is not ready I0314 15:15:36.975097 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0314 15:15:36.975123 1 controller.go:489] The operator is still being initialized I0314 15:15:36.975162 1 controller.go:512] The operator is healthy I0314 15:15:36.975281 1 tasks_processing.go:74] worker 19 stopped. I0314 15:15:36.975297 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 13.149102ms to process 0 records I0314 15:15:36.975305 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 14.535712ms to process 0 records I0314 15:15:36.975312 1 tasks_processing.go:74] worker 13 stopped. I0314 15:15:36.975548 1 tasks_processing.go:74] worker 28 stopped. I0314 15:15:36.976498 1 recorder.go:75] Recording config/infrastructure with fingerprint=b0294a7976e2d552943799da6f7e7c9f2d1ede914c7c3c6f88109ebd6f04217a I0314 15:15:36.976529 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 13.515171ms to process 1 records I0314 15:15:36.976666 1 tasks_processing.go:74] worker 12 stopped. I0314 15:15:36.976761 1 recorder.go:75] Recording config/network with fingerprint=db183ac6d78dd98063888fd2e9862536b9dbecf900e8296e87a2f31179b96d4c I0314 15:15:36.976838 1 gather.go:177] gatherer "clusterconfig" function "networks" took 14.229074ms to process 1 records I0314 15:15:36.977021 1 tasks_processing.go:74] worker 20 stopped. I0314 15:15:36.977024 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=ffdffaa40da7fccc826153edce653de61052132e8421aec7918bafef7adf6f4e I0314 15:15:36.977106 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=ec10dd04e5f866a273513f584ea3a02afa184331e2cfc94e0f65217d4e0dd5bb I0314 15:15:36.977163 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=b0eff420037d8f5ab3fccd4e1d16620763d68d45987fa3587dd817bed7e5e063 I0314 15:15:36.977178 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 12.167916ms to process 3 records I0314 15:15:36.977196 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 47.410953ms to process 0 records I0314 15:15:36.977209 1 tasks_processing.go:74] worker 10 stopped. I0314 15:15:36.979050 1 prometheus_rules.go:88] Prometheus rules successfully created I0314 15:15:36.982843 1 tasks_processing.go:74] worker 5 stopped. I0314 15:15:36.983138 1 recorder.go:75] Recording config/version with fingerprint=25a1c9840517eed78923b6d67bed5adaee1406d606b93e07b4218fb9537d4b38 I0314 15:15:36.983191 1 recorder.go:75] Recording config/id with fingerprint=1b7be1a249477f64e99dd3bd6169caed7b368bee63995ea27a08bcdd319ff506 I0314 15:15:36.983216 1 gather_logs.go:145] no pods in namespace were found I0314 15:15:36.983223 1 gather.go:177] gatherer "clusterconfig" function "version" took 40.888386ms to process 2 records I0314 15:15:36.983298 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 41.224861ms to process 0 records I0314 15:15:36.983327 1 tasks_processing.go:74] worker 3 stopped. I0314 15:15:36.983503 1 tasks_processing.go:74] worker 31 stopped. I0314 15:15:36.983521 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 25.660363ms to process 0 records I0314 15:15:36.983786 1 tasks_processing.go:74] worker 29 stopped. I0314 15:15:36.984358 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=4c7a875f2260a0a042078ec36dcf19d228970553425ce50bb04ffc04c529a783 I0314 15:15:36.984413 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 20.425815ms to process 1 records I0314 15:15:36.985353 1 tasks_processing.go:74] worker 18 stopped. E0314 15:15:36.985981 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27480afac8-bd14-4b06-98e9-03e5bfc89f4f%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:33488->172.30.0.10:53: read: connection refused I0314 15:15:36.986036 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27480afac8-bd14-4b06-98e9-03e5bfc89f4f%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:33488->172.30.0.10:53: read: connection refused I0314 15:15:36.986322 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=1e0b104af7cbf0207d04b5ccf9d81c1fd388ad4ca364e85546a05f03177b19c8 I0314 15:15:36.986667 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=d263c217693e2c891f3eb24032ed43989f81f6f5ff85522d90cd60f67f902237 I0314 15:15:36.986681 1 gather.go:177] gatherer "clusterconfig" function "crds" took 22.782682ms to process 2 records I0314 15:15:36.989042 1 tasks_processing.go:74] worker 22 stopped. E0314 15:15:36.989059 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0314 15:15:36.989068 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0314 15:15:36.989073 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0314 15:15:36.989086 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0314 15:15:36.989128 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0314 15:15:36.989139 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0314 15:15:36.989146 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0314 15:15:36.989154 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0314 15:15:36.989201 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0314 15:15:36.989213 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0314 15:15:36.989223 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 60.676684ms to process 7 records I0314 15:15:36.989390 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0314 15:15:36.994329 1 tasks_processing.go:74] worker 27 stopped. E0314 15:15:36.994388 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0314 15:15:36.994400 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2p1ijtkhngk7nsj2i0lb67l888psn995-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2p1ijtkhngk7nsj2i0lb67l888psn995-primary-cert-bundle-secret" not found I0314 15:15:36.994461 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=327f6841e275deef7e1224e788200742eee7b8a72b62ee63340269d715b0df6b I0314 15:15:36.994500 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 52.237306ms to process 1 records W0314 15:15:36.995204 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:60545->172.30.0.10:53: read: connection refused I0314 15:15:36.995261 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:60545->172.30.0.10:53: read: connection refused I0314 15:15:36.998526 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0314 15:15:36.998536 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0314 15:15:36.998570 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0314 15:15:37.002639 1 base_controller.go:82] Caches are synced for ConfigController I0314 15:15:37.002659 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0314 15:15:37.006316 1 configmapobserver.go:84] configmaps "insights-config" not found I0314 15:15:37.053837 1 tasks_processing.go:74] worker 17 stopped. I0314 15:15:37.053873 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0314 15:15:37.053888 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 95.455965ms to process 1 records I0314 15:15:37.060327 1 base_controller.go:82] Caches are synced for LoggingSyncer I0314 15:15:37.060343 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... W0314 15:15:37.958535 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0314 15:15:37.982069 1 tasks_processing.go:74] worker 15 stopped. I0314 15:15:37.982142 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 1.03984499s to process 0 records I0314 15:15:37.989704 1 tasks_processing.go:74] worker 1 stopped. I0314 15:15:37.989750 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0314 15:15:37.989763 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 1.061565895s to process 1 records I0314 15:15:38.202276 1 gather_cluster_operator_pods_and_events.go:121] Found 12 pods with 14 containers I0314 15:15:38.202288 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1797558 bytes I0314 15:15:38.202523 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-78jrc pod in namespace openshift-dns (previous: false). I0314 15:15:38.413514 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0314 15:15:38.440526 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-78jrc pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-78jrc\" is waiting to start: ContainerCreating" I0314 15:15:38.440544 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-78jrc\" is waiting to start: ContainerCreating" I0314 15:15:38.440552 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-78jrc pod in namespace openshift-dns (previous: false). I0314 15:15:38.590590 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-78jrc pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-78jrc\" is waiting to start: ContainerCreating" I0314 15:15:38.590606 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-78jrc\" is waiting to start: ContainerCreating" I0314 15:15:38.590616 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-zbnk2 pod in namespace openshift-dns (previous: false). I0314 15:15:38.790309 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0314 15:15:38.790326 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-68c745c64-h2q6j pod in namespace openshift-image-registry (previous: false). W0314 15:15:38.957888 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0314 15:15:38.984627 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0314 15:15:38.984643 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-76f6d44bc9-78tpq pod in namespace openshift-image-registry (previous: false). I0314 15:15:38.998208 1 tasks_processing.go:74] worker 24 stopped. I0314 15:15:38.998254 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=f5ee40dfa598d161f816dad4ba43ca8a461c132646a06e77612a66e120508ce9 I0314 15:15:38.998295 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=15722c3c0063dbfcbf0f9da00e58b6bbe5e43e54c2c8b0d85c9c34844d8d99cd I0314 15:15:38.998324 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0314 15:15:38.998348 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=8a320c942b10e1da6654ff86b816e7bc71dd11f11d55c0c254fee58666766e28 I0314 15:15:38.998364 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0314 15:15:38.998387 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=2d445da9b4e88e9c42f8f183a31a5844bf92e49d398b3862a8c3b37595a87350 I0314 15:15:38.998426 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=0935b4716dd755af8acb96c9befd1eb598075b75dbaf21f0e436000b31074371 I0314 15:15:38.998449 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=295fe1fb030f18f9839cea89e9021579bc1379571300a426ae4bbc8e7229d63c I0314 15:15:38.998463 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=0c9bf41cb5dcb0ec5a67c42091eff5a0ba2e1f3b90d6727cd3dd6c02da37125b I0314 15:15:38.998495 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=48d889772f04c4bab09bc62c926048fe1e032725b71f85a1dbe6164a34f15913 I0314 15:15:38.998505 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0314 15:15:38.998521 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=147a89e319dc4dc83bbc4295344f446e1bab2b7c290391b789d7020665626a0a I0314 15:15:38.998532 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0314 15:15:38.998547 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=ad361a5867474e9d8f0911f69108cdece04ad8e90b5ea8fa0c66398258e3d08f I0314 15:15:38.998556 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0314 15:15:38.998576 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=0f501d7f42d5a16f757642586922be413444551b8cf763b1fa5101caf0624e26 I0314 15:15:38.998585 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0314 15:15:38.998600 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=c7328c046aeb353ceebeaeeed696baa23719a1ae9f3985779640adcd9be5d06c I0314 15:15:38.998772 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=2caf96c2433a600fdb1c6e9de4d58ff4487c3d5d768fedf8cf9ed25d0377af48 I0314 15:15:38.998784 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0314 15:15:38.998792 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0314 15:15:38.998814 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0314 15:15:38.998835 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=cd05e0becc606ef9e7022b4053654f567394b50e8d3ee42059639e3c4506a99c I0314 15:15:38.998860 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=03c28e43f2feb05da1018502403ba8cd684ba5c26f772406c86c5fb7eb3e8beb I0314 15:15:38.998870 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0314 15:15:38.998886 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=849dafe5a7449c4bee682025def5307a75966218ef24177f052e2c0cd07f51c1 I0314 15:15:38.998896 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0314 15:15:38.998909 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=d246355b603e89e99b5db2016a63b083575d51236ee25546e150d1480fe7fa1f I0314 15:15:38.998924 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=7225f179184e74d8a9b84eb7e73d6198dcf5a5e483fb80e4b2b64b78a67e22f9 I0314 15:15:38.998937 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=b9ef662fd08b15d554bb222c914068efa653e8c3755799de5513d1516f79b146 I0314 15:15:38.998952 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=08d8ae89f471ef4935b5fc97a5047becdbe1476ac67c849cd5fadc2d41a4bbbe I0314 15:15:38.998972 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=5e3c24b87566c2411679da08e67532bc475ef1cbfb2a38fd02caae81d30518f7 I0314 15:15:38.998981 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0314 15:15:38.999007 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=baf785c29861ef2e85384f39ccb5121fed099ab0a1ff7330ca5982a17bcdf4f9 I0314 15:15:38.999024 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0314 15:15:38.999032 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0314 15:15:38.999038 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.06874925s to process 36 records I0314 15:15:39.186691 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0314 15:15:39.186735 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-76f6d44bc9-8tszm pod in namespace openshift-image-registry (previous: false). I0314 15:15:39.389426 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-76f6d44bc9-8tszm pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-76f6d44bc9-8tszm\" is waiting to start: ContainerCreating" I0314 15:15:39.389443 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-76f6d44bc9-8tszm\" is waiting to start: ContainerCreating" I0314 15:15:39.389453 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-hmc4g pod in namespace openshift-image-registry (previous: false). I0314 15:15:39.593332 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0314 15:15:39.593350 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-56fbfbd9b-hfpwr pod in namespace openshift-ingress (previous: false). I0314 15:15:39.789758 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-56fbfbd9b-hfpwr pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-56fbfbd9b-hfpwr\" is waiting to start: ContainerCreating" I0314 15:15:39.789779 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-56fbfbd9b-hfpwr\" is waiting to start: ContainerCreating" I0314 15:15:39.789793 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-56fbfbd9b-s6g6j pod in namespace openshift-ingress (previous: false). W0314 15:15:39.959266 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0314 15:15:39.984437 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0314 15:15:39.984517 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-796564cf8c-wqsfg pod in namespace openshift-ingress (previous: false). I0314 15:15:40.189863 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-796564cf8c-wqsfg pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-796564cf8c-wqsfg\" is waiting to start: ContainerCreating" I0314 15:15:40.189882 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-796564cf8c-wqsfg\" is waiting to start: ContainerCreating" I0314 15:15:40.189895 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-4hmrp pod in namespace openshift-ingress-canary (previous: false). I0314 15:15:40.390958 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-4hmrp pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-4hmrp\" is waiting to start: ContainerCreating" I0314 15:15:40.390974 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-4hmrp\" is waiting to start: ContainerCreating" I0314 15:15:40.390987 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for migrator container migrator-7d5f866c57-vzrnk pod in namespace openshift-kube-storage-version-migrator (previous: false). I0314 15:15:40.591098 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for graceful-termination container migrator-7d5f866c57-vzrnk pod in namespace openshift-kube-storage-version-migrator (previous: false). I0314 15:15:40.790607 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-storage-version-migrator-operator container kube-storage-version-migrator-operator-74848b4cb9-l2wlj pod in namespace openshift-kube-storage-version-migrator-operator (previous: false). W0314 15:15:40.958363 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0314 15:15:40.993978 1 tasks_processing.go:74] worker 7 stopped. I0314 15:15:40.994070 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=1a1db123aaeaeb5b8a7cc1ca29b31bbb8bdee2b213c9182dbfbcce6622b3591c I0314 15:15:40.994096 1 recorder.go:75] Recording events/openshift-dns with fingerprint=142230bffb79e8cfeb49855ff675f60c45007645d63af2e3618c8b75c6c700d6 I0314 15:15:40.994162 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=c7c78faf2568fa9bc2240f1bb8bbac7fdc595c32993c8e0f8abec21029af37c8 I0314 15:15:40.994189 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=2119b7cc03423c1be4a1dcc591476dd480d376ca1aad48a883a500164b9509c3 I0314 15:15:40.994230 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=e9a5e6f39c97be34e5952e778a7efc2a7ecc11b30d09fefa160aafc1742b1e1e I0314 15:15:40.994245 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=85a91b4d1e71b9682c2d9d9222f7c2dfe838a3d2e00008247d786cd4aff74bb6 I0314 15:15:40.994262 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator with fingerprint=5de7f8555c6886aa6cf4bc7acc31c9ab610bab2b77e024fa507d022164f7ef1c I0314 15:15:40.994320 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator-operator with fingerprint=d719dedb1a10fa89c1b121a5d029d1e409610fe7ad6338a7fa8afdbc0d1258b5 I0314 15:15:40.994342 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-vzrnk/migrator_current.log with fingerprint=a8ebed3a59f6667e165d91ed9cf1c913e5f9be7d1bc94c84fd27bebef6369157 I0314 15:15:40.994346 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-vzrnk/graceful-termination_current.log with fingerprint=3f92f53096d4b5b9df2b7014b377c7109c6038969b05b752c37b7bef7d54b257 I0314 15:15:40.994414 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator-operator/logs/kube-storage-version-migrator-operator-74848b4cb9-l2wlj/kube-storage-version-migrator-operator_current.log with fingerprint=598bcf964bb13decb1a16d92b7921de5cffc8c51add0823841682d7dff79c201 I0314 15:15:40.994424 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 4.03118374s to process 11 records W0314 15:15:41.964466 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0314 15:15:41.964572 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0314 15:15:41.964628 1 tasks_processing.go:74] worker 11 stopped. E0314 15:15:41.964666 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0314 15:15:41.964700 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0314 15:15:41.964764 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0314 15:15:41.964797 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.035313696s to process 1 records I0314 15:15:49.320606 1 configmapobserver.go:84] configmaps "insights-config" not found I0314 15:15:49.565088 1 tasks_processing.go:74] worker 6 stopped. I0314 15:15:49.565126 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0314 15:15:49.565141 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.635760449s to process 1 records I0314 15:15:50.348103 1 tasks_processing.go:74] worker 26 stopped. I0314 15:15:50.348370 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=e03550ff5cb13d4cd718a41a9755571c983753e7ec8a789b8fb921884bbb0f64 I0314 15:15:50.348387 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.406040062s to process 1 records E0314 15:15:50.348452 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.42s with: function \"support_secret\" failed with an error, function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0314 15:15:50.349554 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "support_secret" failed with an error, function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0314 15:15:50.349569 1 periodic.go:209] Running workloads gatherer I0314 15:15:50.349583 1 tasks_processing.go:45] number of workers: 2 I0314 15:15:50.349591 1 tasks_processing.go:69] worker 1 listening for tasks. I0314 15:15:50.349596 1 tasks_processing.go:71] worker 1 working on workload_info task. I0314 15:15:50.349605 1 tasks_processing.go:69] worker 0 listening for tasks. I0314 15:15:50.349623 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0314 15:15:50.371828 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 20s for image data I0314 15:15:50.376235 1 tasks_processing.go:74] worker 0 stopped. I0314 15:15:50.376248 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 26.599478ms to process 0 records I0314 15:15:50.383999 1 gather_workloads_info.go:387] No image sha256:eff3318d5ac1d3a141221b536233305f54b3334fab56e5f8ba20ef8a9ed21429 (13ms) I0314 15:15:50.392375 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (8ms) I0314 15:15:50.405376 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (13ms) I0314 15:15:50.413927 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (9ms) I0314 15:15:50.422071 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (8ms) I0314 15:15:50.430081 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (8ms) I0314 15:15:50.439293 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (9ms) I0314 15:15:50.448401 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (9ms) I0314 15:15:50.456690 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (8ms) I0314 15:15:50.464531 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (8ms) I0314 15:15:50.479904 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (15ms) I0314 15:15:50.580948 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (101ms) I0314 15:15:50.680804 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (100ms) I0314 15:15:50.781022 1 gather_workloads_info.go:387] No image sha256:da4f8a01ef5d0607cf76fa7b219fafdb6020cb4cf9a0a88cfe76d292c1a561fc (100ms) I0314 15:15:50.860729 1 configmapobserver.go:84] configmaps "insights-config" not found I0314 15:15:50.881414 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (100ms) I0314 15:15:50.981688 1 gather_workloads_info.go:387] No image sha256:fac526ab9335a9a8decb028a30673f4ae718deae8da114f36103b6438ba6458f (100ms) I0314 15:15:51.055676 1 configmapobserver.go:84] configmaps "insights-config" not found I0314 15:15:51.080217 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (99ms) I0314 15:15:51.180404 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0314 15:15:51.281249 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (101ms) I0314 15:15:51.380336 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (99ms) I0314 15:15:51.481365 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (101ms) I0314 15:15:51.581026 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0314 15:15:51.681918 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (101ms) I0314 15:15:51.781998 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (100ms) I0314 15:15:51.881991 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0314 15:15:51.981262 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (99ms) I0314 15:15:52.080203 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (99ms) I0314 15:15:52.180433 1 gather_workloads_info.go:387] No image sha256:d3209a996e5ce362136e70a0bcf06d89973905de73e234b644d8ae8886d067f1 (100ms) I0314 15:15:52.280703 1 gather_workloads_info.go:387] No image sha256:ef2bfb546cd8aac31b7671585f718f44b268ed43a9c8562016ab741558a7dfa1 (100ms) I0314 15:15:52.381051 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (100ms) I0314 15:15:52.480755 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (100ms) I0314 15:15:52.581166 1 gather_workloads_info.go:387] No image sha256:fab570ecdc9c3620b032bc4c5b44240f5ebac0bb9e48dfeea32bcd36eb462e5a (100ms) I0314 15:15:52.680178 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (99ms) I0314 15:15:52.780449 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (100ms) I0314 15:15:52.882276 1 gather_workloads_info.go:387] No image sha256:a505dd5b62d744c2cfda2666b5e21e0a4704354a3114488bd3291456085e1e33 (102ms) I0314 15:15:52.981998 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (100ms) I0314 15:15:53.081095 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (99ms) I0314 15:15:53.181187 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0314 15:15:53.283334 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (102ms) I0314 15:15:53.283365 1 tasks_processing.go:74] worker 1 stopped. E0314 15:15:53.283377 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0314 15:15:53.283644 1 recorder.go:75] Recording config/workload_info with fingerprint=cd6fdb805cd03cb1c16c984039304ba456d7331fd3957a4410d81e7caf7e195f I0314 15:15:53.283659 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.933761147s to process 1 records E0314 15:15:53.283681 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.934s with: function \"workload_info\" failed with an error" I0314 15:15:53.284804 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0314 15:15:53.284818 1 periodic.go:209] Running conditional gatherer I0314 15:15:53.290988 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0314 15:15:53.297746 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.12:35635->172.30.0.10:53: read: connection refused E0314 15:15:53.297981 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0314 15:15:53.298036 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0314 15:15:53.303298 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0314 15:15:53.303311 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0314 15:15:53.303316 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0314 15:15:53.303319 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0314 15:15:53.303322 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0314 15:15:53.303325 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0314 15:15:53.303328 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0314 15:15:53.303332 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0314 15:15:53.303336 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0314 15:15:53.303339 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0314 15:15:53.303351 1 tasks_processing.go:45] number of workers: 3 I0314 15:15:53.303363 1 tasks_processing.go:69] worker 2 listening for tasks. I0314 15:15:53.303367 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0314 15:15:53.303371 1 tasks_processing.go:69] worker 0 listening for tasks. I0314 15:15:53.303376 1 tasks_processing.go:69] worker 1 listening for tasks. I0314 15:15:53.303394 1 tasks_processing.go:74] worker 1 stopped. I0314 15:15:53.303403 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0314 15:15:53.303415 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0314 15:15:53.303439 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0314 15:15:53.303450 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 700ns to process 1 records I0314 15:15:53.303479 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0314 15:15:53.303487 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.323µs to process 1 records I0314 15:15:53.303492 1 tasks_processing.go:74] worker 0 stopped. I0314 15:15:53.303641 1 tasks_processing.go:74] worker 2 stopped. I0314 15:15:53.303652 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 230.707µs to process 0 records I0314 15:15:53.303676 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.12:35635->172.30.0.10:53: read: connection refused I0314 15:15:53.303692 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0314 15:15:53.325786 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=f1d3617cf6fd282f00d8b2db5de2623c781b3c5d619179d3149ddb039349d4bb I0314 15:15:53.325928 1 diskrecorder.go:70] Writing 102 records to /var/lib/insights-operator/insights-2026-03-14-151553.tar.gz I0314 15:15:53.332273 1 diskrecorder.go:51] Wrote 102 records to disk in 6ms I0314 15:15:53.332299 1 periodic.go:278] Gathering cluster info every 2h0m0s I0314 15:15:53.332312 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0314 15:16:02.058117 1 configmapobserver.go:84] configmaps "insights-config" not found I0314 15:17:05.935023 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="aeb22b8faa06bc6ce52f8069a85c4ee2b8e4c2cf805a51d1e12946da5680a6ec") W0314 15:17:05.935055 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0314 15:17:05.935113 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="db7e2fb3aecfe0b83b47ca18c75bb114b753716d5ac97a032b13f805fb8c13ea") I0314 15:17:05.935114 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0314 15:17:05.935127 1 base_controller.go:181] Shutting down LoggingSyncer ... I0314 15:17:05.935134 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0314 15:17:05.935168 1 base_controller.go:181] Shutting down ConfigController ... I0314 15:17:05.935200 1 periodic.go:170] Shutting down I0314 15:17:05.935200 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0314 15:17:05.935210 1 base_controller.go:113] All ConfigController workers have been terminated