W0521 19:50:59.052684 1 cmd.go:257] Using insecure, self-signed certificates I0521 19:50:59.492328 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:50:59.494071 1 observer_polling.go:159] Starting file observer I0521 19:51:00.741774 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0521 19:51:00.741969 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0521 19:51:00.742534 1 secure_serving.go:57] Forcing use of http/1.1 only W0521 19:51:00.742553 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0521 19:51:00.742558 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0521 19:51:00.742564 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0521 19:51:00.742566 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0521 19:51:00.742569 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0521 19:51:00.742572 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0521 19:51:00.742617 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0521 19:51:00.751344 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0521 19:51:00.751363 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"d6389c6f-727b-46cd-ab19-c2755d6b256c", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0521 19:51:00.753465 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0521 19:51:00.753479 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0521 19:51:00.753490 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0521 19:51:00.753512 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0521 19:51:00.753533 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:51:00.753534 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:51:00.753774 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-3545912871/tls.crt::/tmp/serving-cert-3545912871/tls.key" I0521 19:51:00.754059 1 secure_serving.go:213] Serving securely on [::]:8443 I0521 19:51:00.754085 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0521 19:51:00.779331 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0521 19:51:00.779367 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:51:00.779465 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0521 19:51:00.791747 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:51:00.791771 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:51:00.796847 1 secretconfigobserver.go:119] support secret does not exist I0521 19:51:00.802032 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:51:00.805994 1 secretconfigobserver.go:119] support secret does not exist I0521 19:51:00.809137 1 recorder.go:161] Pruning old reports every 5h15m46s, max age is 288h0m0s I0521 19:51:00.814081 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0521 19:51:00.814096 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0521 19:51:00.814113 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0521 19:51:00.814118 1 periodic.go:209] Running clusterconfig gatherer I0521 19:51:00.814127 1 insightsreport.go:296] Starting report retriever I0521 19:51:00.814136 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0521 19:51:00.814162 1 tasks_processing.go:45] number of workers: 64 I0521 19:51:00.814184 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:51:00.814194 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:51:00.814196 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:51:00.814205 1 tasks_processing.go:71] worker 1 working on openstack_dataplanenodesets task. I0521 19:51:00.814208 1 tasks_processing.go:71] worker 0 working on storage_cluster task. I0521 19:51:00.814210 1 tasks_processing.go:69] worker 37 listening for tasks. I0521 19:51:00.814220 1 tasks_processing.go:69] worker 32 listening for tasks. I0521 19:51:00.814230 1 tasks_processing.go:69] worker 33 listening for tasks. I0521 19:51:00.814227 1 tasks_processing.go:69] worker 31 listening for tasks. I0521 19:51:00.814231 1 tasks_processing.go:69] worker 3 listening for tasks. I0521 19:51:00.814238 1 tasks_processing.go:69] worker 4 listening for tasks. I0521 19:51:00.814239 1 tasks_processing.go:69] worker 26 listening for tasks. I0521 19:51:00.814243 1 tasks_processing.go:69] worker 5 listening for tasks. I0521 19:51:00.814259 1 tasks_processing.go:69] worker 51 listening for tasks. I0521 19:51:00.814268 1 tasks_processing.go:69] worker 34 listening for tasks. I0521 19:51:00.814269 1 tasks_processing.go:69] worker 27 listening for tasks. I0521 19:51:00.814270 1 tasks_processing.go:69] worker 6 listening for tasks. I0521 19:51:00.814277 1 tasks_processing.go:69] worker 45 listening for tasks. I0521 19:51:00.814271 1 tasks_processing.go:69] worker 44 listening for tasks. I0521 19:51:00.814284 1 tasks_processing.go:69] worker 28 listening for tasks. I0521 19:51:00.814277 1 tasks_processing.go:69] worker 7 listening for tasks. I0521 19:51:00.814288 1 tasks_processing.go:69] worker 36 listening for tasks. I0521 19:51:00.814279 1 tasks_processing.go:69] worker 35 listening for tasks. I0521 19:51:00.814281 1 tasks_processing.go:69] worker 58 listening for tasks. I0521 19:51:00.814297 1 tasks_processing.go:69] worker 39 listening for tasks. I0521 19:51:00.814288 1 tasks_processing.go:69] worker 8 listening for tasks. I0521 19:51:00.814301 1 tasks_processing.go:69] worker 47 listening for tasks. I0521 19:51:00.814304 1 tasks_processing.go:69] worker 30 listening for tasks. I0521 19:51:00.814306 1 tasks_processing.go:69] worker 53 listening for tasks. I0521 19:51:00.814312 1 tasks_processing.go:69] worker 48 listening for tasks. I0521 19:51:00.814284 1 tasks_processing.go:69] worker 38 listening for tasks. I0521 19:51:00.814310 1 tasks_processing.go:69] worker 16 listening for tasks. I0521 19:51:00.814321 1 tasks_processing.go:69] worker 23 listening for tasks. I0521 19:51:00.814324 1 tasks_processing.go:69] worker 49 listening for tasks. I0521 19:51:00.814325 1 tasks_processing.go:69] worker 54 listening for tasks. I0521 19:51:00.814329 1 tasks_processing.go:69] worker 13 listening for tasks. I0521 19:51:00.814330 1 tasks_processing.go:69] worker 61 listening for tasks. I0521 19:51:00.814330 1 tasks_processing.go:69] worker 24 listening for tasks. I0521 19:51:00.814332 1 tasks_processing.go:69] worker 19 listening for tasks. I0521 19:51:00.814335 1 tasks_processing.go:69] worker 50 listening for tasks. I0521 19:51:00.814337 1 tasks_processing.go:69] worker 14 listening for tasks. I0521 19:51:00.814336 1 tasks_processing.go:69] worker 20 listening for tasks. I0521 19:51:00.814340 1 tasks_processing.go:69] worker 59 listening for tasks. I0521 19:51:00.814345 1 tasks_processing.go:69] worker 62 listening for tasks. I0521 19:51:00.814342 1 tasks_processing.go:69] worker 60 listening for tasks. I0521 19:51:00.814344 1 tasks_processing.go:69] worker 63 listening for tasks. I0521 19:51:00.814349 1 tasks_processing.go:69] worker 11 listening for tasks. I0521 19:51:00.814295 1 tasks_processing.go:69] worker 17 listening for tasks. I0521 19:51:00.814356 1 tasks_processing.go:71] worker 2 working on authentication task. I0521 19:51:00.814352 1 tasks_processing.go:69] worker 12 listening for tasks. I0521 19:51:00.814298 1 tasks_processing.go:69] worker 29 listening for tasks. I0521 19:51:00.814367 1 tasks_processing.go:69] worker 9 listening for tasks. I0521 19:51:00.814298 1 tasks_processing.go:69] worker 52 listening for tasks. I0521 19:51:00.814392 1 tasks_processing.go:71] worker 52 working on sap_pods task. I0521 19:51:00.814394 1 tasks_processing.go:71] worker 9 working on openshift_machine_api_events task. I0521 19:51:00.814301 1 tasks_processing.go:69] worker 18 listening for tasks. I0521 19:51:00.814307 1 tasks_processing.go:69] worker 40 listening for tasks. I0521 19:51:00.814425 1 tasks_processing.go:71] worker 23 working on image_registries task. I0521 19:51:00.814428 1 tasks_processing.go:71] worker 37 working on overlapping_namespace_uids task. I0521 19:51:00.814201 1 tasks_processing.go:69] worker 25 listening for tasks. I0521 19:51:00.814433 1 tasks_processing.go:71] worker 14 working on infrastructures task. I0521 19:51:00.814437 1 tasks_processing.go:71] worker 61 working on oauths task. I0521 19:51:00.814439 1 tasks_processing.go:71] worker 32 working on image task. I0521 19:51:00.814445 1 tasks_processing.go:71] worker 53 working on tsdb_status task. W0521 19:51:00.814473 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:51:00.814486 1 tasks_processing.go:71] worker 59 working on mutating_webhook_configurations task. I0521 19:51:00.814491 1 tasks_processing.go:71] worker 53 working on openshift_logging task. I0521 19:51:00.814493 1 tasks_processing.go:71] worker 25 working on version task. I0521 19:51:00.814506 1 tasks_processing.go:71] worker 17 working on sap_datahubs task. I0521 19:51:00.814520 1 tasks_processing.go:71] worker 18 working on container_runtime_configs task. I0521 19:51:00.814537 1 tasks_processing.go:71] worker 62 working on clusterroles task. I0521 19:51:00.814434 1 tasks_processing.go:71] worker 13 working on image_pruners task. I0521 19:51:00.814552 1 tasks_processing.go:71] worker 33 working on nodes task. I0521 19:51:00.814565 1 tasks_processing.go:71] worker 5 working on cluster_apiserver task. I0521 19:51:00.814586 1 tasks_processing.go:71] worker 51 working on monitoring_persistent_volumes task. I0521 19:51:00.814594 1 tasks_processing.go:71] worker 24 working on ceph_cluster task. I0521 19:51:00.814657 1 tasks_processing.go:71] worker 38 working on operators_pods_and_events task. I0521 19:51:00.814683 1 tasks_processing.go:71] worker 63 working on openstack_controlplanes task. I0521 19:51:00.814315 1 tasks_processing.go:69] worker 41 listening for tasks. I0521 19:51:00.814717 1 tasks_processing.go:71] worker 12 working on cost_management_metrics_configs task. I0521 19:51:00.814292 1 tasks_processing.go:69] worker 46 listening for tasks. I0521 19:51:00.814722 1 tasks_processing.go:71] worker 40 working on networks task. I0521 19:51:00.814722 1 tasks_processing.go:71] worker 16 working on container_images task. I0521 19:51:00.814729 1 tasks_processing.go:71] worker 29 working on operators task. I0521 19:51:00.814734 1 tasks_processing.go:71] worker 28 working on proxies task. I0521 19:51:00.814735 1 tasks_processing.go:71] worker 45 working on qemu_kubevirt_launcher_logs task. I0521 19:51:00.814738 1 tasks_processing.go:71] worker 35 working on machine_config_pools task. I0521 19:51:00.814744 1 tasks_processing.go:71] worker 4 working on jaegers task. I0521 19:51:00.814748 1 tasks_processing.go:71] worker 8 working on config_maps task. I0521 19:51:00.814750 1 tasks_processing.go:71] worker 47 working on olm_operators task. I0521 19:51:00.815470 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 37.325µs to process 0 records I0521 19:51:00.814734 1 tasks_processing.go:71] worker 31 working on machine_autoscalers task. I0521 19:51:00.814440 1 tasks_processing.go:71] worker 48 working on machine_configs task. I0521 19:51:00.815499 1 tasks_processing.go:71] worker 46 working on schedulers task. I0521 19:51:00.814739 1 tasks_processing.go:71] worker 3 working on nodenetworkconfigurationpolicies task. I0521 19:51:00.814317 1 tasks_processing.go:69] worker 56 listening for tasks. I0521 19:51:00.816377 1 tasks_processing.go:71] worker 56 working on ingress_certificates task. I0521 19:51:00.816396 1 tasks_processing.go:71] worker 36 working on certificate_signing_requests task. I0521 19:51:00.814323 1 tasks_processing.go:69] worker 42 listening for tasks. I0521 19:51:00.814318 1 tasks_processing.go:69] worker 57 listening for tasks. I0521 19:51:00.814321 1 tasks_processing.go:69] worker 21 listening for tasks. I0521 19:51:00.814331 1 tasks_processing.go:69] worker 43 listening for tasks. I0521 19:51:00.814342 1 tasks_processing.go:69] worker 15 listening for tasks. I0521 19:51:00.814346 1 tasks_processing.go:69] worker 10 listening for tasks. I0521 19:51:00.814422 1 tasks_processing.go:71] worker 30 working on pdbs task. I0521 19:51:00.816556 1 tasks_processing.go:71] worker 10 working on openstack_version task. I0521 19:51:00.814423 1 tasks_processing.go:71] worker 19 working on active_alerts task. I0521 19:51:00.816600 1 tasks_processing.go:71] worker 21 working on aggregated_monitoring_cr_names task. I0521 19:51:00.816678 1 tasks_processing.go:71] worker 42 working on validating_webhook_configurations task. I0521 19:51:00.816712 1 tasks_processing.go:71] worker 15 working on install_plans task. W0521 19:51:00.816763 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:51:00.816785 1 tasks_processing.go:71] worker 19 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0521 19:51:00.814423 1 tasks_processing.go:71] worker 60 working on storage_classes task. I0521 19:51:00.816849 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 181.08µs to process 0 records I0521 19:51:00.816733 1 tasks_processing.go:71] worker 43 working on dvo_metrics task. I0521 19:51:00.814428 1 tasks_processing.go:71] worker 50 working on lokistack task. I0521 19:51:00.814427 1 tasks_processing.go:71] worker 49 working on node_logs task. I0521 19:51:00.814431 1 tasks_processing.go:71] worker 54 working on pod_network_connectivity_checks task. I0521 19:51:00.814436 1 tasks_processing.go:71] worker 6 working on machine_healthchecks task. I0521 19:51:00.814482 1 tasks_processing.go:71] worker 20 working on crds task. I0521 19:51:00.814720 1 tasks_processing.go:71] worker 11 working on support_secret task. I0521 19:51:00.814731 1 tasks_processing.go:71] worker 44 working on service_accounts task. I0521 19:51:00.814733 1 tasks_processing.go:71] worker 39 working on feature_gates task. I0521 19:51:00.814749 1 tasks_processing.go:71] worker 58 working on ingress task. I0521 19:51:00.814753 1 tasks_processing.go:71] worker 26 working on silenced_alerts task. I0521 19:51:00.815622 1 tasks_processing.go:71] worker 41 working on openstack_dataplanedeployments task. I0521 19:51:00.815687 1 tasks_processing.go:71] worker 7 working on sap_config task. I0521 19:51:00.815749 1 tasks_processing.go:71] worker 34 working on nodenetworkstates task. I0521 19:51:00.814320 1 tasks_processing.go:69] worker 55 listening for tasks. I0521 19:51:00.815764 1 tasks_processing.go:71] worker 27 working on metrics task. I0521 19:51:00.816747 1 tasks_processing.go:71] worker 57 working on machine_sets task. W0521 19:51:00.817491 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:51:00.817504 1 tasks_processing.go:74] worker 27 stopped. I0521 19:51:00.817511 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 29.36µs to process 0 records I0521 19:51:00.814309 1 tasks_processing.go:69] worker 22 listening for tasks. I0521 19:51:00.817531 1 tasks_processing.go:74] worker 22 stopped. I0521 19:51:00.818058 1 tasks_processing.go:74] worker 1 stopped. I0521 19:51:00.818069 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 3.846138ms to process 0 records I0521 19:51:00.818096 1 tasks_processing.go:74] worker 0 stopped. I0521 19:51:00.818106 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 3.880022ms to process 0 records W0521 19:51:00.818238 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:51:00.818274 1 tasks_processing.go:74] worker 26 stopped. I0521 19:51:00.818285 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 927.458µs to process 0 records I0521 19:51:00.818419 1 tasks_processing.go:71] worker 55 working on machines task. I0521 19:51:00.819348 1 tasks_processing.go:74] worker 53 stopped. I0521 19:51:00.819363 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 4.844876ms to process 0 records I0521 19:51:00.819371 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 4.81708ms to process 0 records I0521 19:51:00.819378 1 tasks_processing.go:74] worker 18 stopped. I0521 19:51:00.822381 1 tasks_processing.go:74] worker 63 stopped. I0521 19:51:00.822394 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 7.688944ms to process 0 records I0521 19:51:00.822405 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 7.873122ms to process 0 records I0521 19:51:00.822416 1 tasks_processing.go:74] worker 17 stopped. I0521 19:51:00.825277 1 tasks_processing.go:74] worker 32 stopped. I0521 19:51:00.825395 1 recorder.go:75] Recording config/image with fingerprint=dcbe485a245b456698fb65e67cf8a9b9e2b4a2b13eb9ea50f12d7270b6d8e06d I0521 19:51:00.825412 1 gather.go:177] gatherer "clusterconfig" function "image" took 10.824623ms to process 1 records I0521 19:51:00.829114 1 tasks_processing.go:74] worker 24 stopped. I0521 19:51:00.829126 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 14.512636ms to process 0 records I0521 19:51:00.829284 1 tasks_processing.go:74] worker 31 stopped. I0521 19:51:00.829299 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 13.448847ms to process 0 records I0521 19:51:00.829409 1 tasks_processing.go:74] worker 14 stopped. I0521 19:51:00.829882 1 recorder.go:75] Recording config/infrastructure with fingerprint=172c62829bc6c00d3f3406b567fb3ceea988f22f10649641afe941cd5eea4f3c I0521 19:51:00.829894 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 14.961152ms to process 1 records I0521 19:51:00.829980 1 tasks_processing.go:74] worker 2 stopped. I0521 19:51:00.830063 1 recorder.go:75] Recording config/authentication with fingerprint=a645daabcd0df660d5d7c969ff282e3d6d84807e7945873f90e22bc9fb5ea393 I0521 19:51:00.830072 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 15.10259ms to process 1 records I0521 19:51:00.830154 1 tasks_processing.go:74] worker 61 stopped. I0521 19:51:00.830230 1 recorder.go:75] Recording config/oauth with fingerprint=41e773a574722974efd41be062e277b1a3ceedc90a76e4290b0d541c1511f69d I0521 19:51:00.830239 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 15.05549ms to process 1 records I0521 19:51:00.830327 1 tasks_processing.go:74] worker 40 stopped. I0521 19:51:00.830359 1 recorder.go:75] Recording config/network with fingerprint=93a8f09d2e67a1400c3d00c4a32983ea0cc08922b5188c9eddf34fe90e865c9b I0521 19:51:00.830367 1 gather.go:177] gatherer "clusterconfig" function "networks" took 14.787164ms to process 1 records I0521 19:51:00.832112 1 tasks_processing.go:74] worker 9 stopped. I0521 19:51:00.832125 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 17.701554ms to process 0 records I0521 19:51:00.838155 1 tasks_processing.go:74] worker 12 stopped. I0521 19:51:00.838168 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 23.4266ms to process 0 records I0521 19:51:00.838181 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 21.982972ms to process 0 records I0521 19:51:00.838188 1 tasks_processing.go:74] worker 3 stopped. I0521 19:51:00.838337 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0521 19:51:00.838353 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:51:00.838357 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0521 19:51:00.838361 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:51:00.838375 1 controller.go:489] The operator is still being initialized I0521 19:51:00.838380 1 controller.go:512] The operator is healthy I0521 19:51:00.838389 1 tasks_processing.go:74] worker 28 stopped. I0521 19:51:00.838444 1 recorder.go:75] Recording config/proxy with fingerprint=6ed1ed49f7a4f9d391e9e9b798e06fd95277b9534877d668ec4d10c350238dbc I0521 19:51:00.838455 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 23.642878ms to process 1 records I0521 19:51:00.838593 1 recorder.go:75] Recording config/apiserver with fingerprint=4795709b32ab73abac512111e937b5df01ac69a96d6e6d1e406a6e0ed4b4d004 I0521 19:51:00.838603 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 23.835573ms to process 1 records I0521 19:51:00.838609 1 tasks_processing.go:74] worker 5 stopped. I0521 19:51:00.839667 1 tasks_processing.go:74] worker 46 stopped. I0521 19:51:00.839734 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=032072d702afa908d16a30d77f5e5d039bb65ec8320df198d17aaaba8fc9fec7 I0521 19:51:00.839746 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 23.593891ms to process 1 records I0521 19:51:00.839993 1 tasks_processing.go:74] worker 37 stopped. I0521 19:51:00.840012 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0521 19:51:00.840021 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 25.550655ms to process 1 records I0521 19:51:00.846043 1 tasks_processing.go:74] worker 57 stopped. I0521 19:51:00.846056 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 28.553458ms to process 0 records I0521 19:51:00.846074 1 tasks_processing.go:74] worker 51 stopped. I0521 19:51:00.846081 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 31.477633ms to process 0 records I0521 19:51:00.846088 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 31.672788ms to process 0 records E0521 19:51:00.846093 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0521 19:51:00.846099 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 29.090148ms to process 0 records I0521 19:51:00.846101 1 tasks_processing.go:74] worker 52 stopped. I0521 19:51:00.846107 1 tasks_processing.go:74] worker 54 stopped. I0521 19:51:00.850366 1 tasks_processing.go:74] worker 4 stopped. I0521 19:51:00.850380 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 35.612392ms to process 0 records I0521 19:51:00.850401 1 tasks_processing.go:74] worker 50 stopped. I0521 19:51:00.850413 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 33.493553ms to process 0 records I0521 19:51:00.850546 1 tasks_processing.go:74] worker 6 stopped. E0521 19:51:00.850555 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0521 19:51:00.850563 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 33.355785ms to process 0 records I0521 19:51:00.850759 1 tasks_processing.go:74] worker 13 stopped. I0521 19:51:00.850949 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=a897d1f6f992f7ce46f4f0962cadab3d788b8f60777e70bba1db99a5fe1ea419 I0521 19:51:00.850962 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 36.202319ms to process 1 records I0521 19:51:00.854349 1 tasks_processing.go:74] worker 41 stopped. I0521 19:51:00.854360 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 36.999473ms to process 0 records I0521 19:51:00.854366 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 36.934843ms to process 0 records I0521 19:51:00.854371 1 tasks_processing.go:74] worker 34 stopped. I0521 19:51:00.854377 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:51:00.854385 1 tasks_processing.go:74] worker 55 stopped. E0521 19:51:00.854391 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0521 19:51:00.854398 1 gather.go:177] gatherer "clusterconfig" function "machines" took 35.956397ms to process 0 records I0521 19:51:00.854408 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:51:00.854414 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0521 19:51:00.854483 1 tasks_processing.go:74] worker 7 stopped. I0521 19:51:00.854498 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 37.092067ms to process 0 records I0521 19:51:00.854827 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 37.917848ms to process 0 records I0521 19:51:00.854842 1 tasks_processing.go:74] worker 10 stopped. I0521 19:51:00.855015 1 tasks_processing.go:74] worker 58 stopped. I0521 19:51:00.855315 1 recorder.go:75] Recording config/ingress with fingerprint=1d03466fcf9f97432aba03c52d9c347d7ab09c0e6618e8d47508be15351a9d1e I0521 19:51:00.855335 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 37.670339ms to process 1 records I0521 19:51:00.855741 1 tasks_processing.go:74] worker 33 stopped. I0521 19:51:00.856743 1 recorder.go:75] Recording config/node/ip-10-0-0-251.ec2.internal with fingerprint=22298dcad6cc293c1b50c498b30a20615ce91c9627400d50eff94a3699d024f2 I0521 19:51:00.856957 1 recorder.go:75] Recording config/node/ip-10-0-1-122.ec2.internal with fingerprint=06e78e3e280003f73385a226fe5be8d922811074e0e49c805ebcff2c6a0024d3 I0521 19:51:00.857107 1 recorder.go:75] Recording config/node/ip-10-0-2-223.ec2.internal with fingerprint=01a3194597dbc38223a2c38cec764a71112f38939a21c8c402c54b4030fdf323 I0521 19:51:00.857512 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 41.177403ms to process 3 records I0521 19:51:00.857627 1 tasks_processing.go:74] worker 23 stopped. I0521 19:51:00.858028 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=94528aa71aed1e346a3ada73e946e8f1366da57980e5203f5998f62bb802e479 I0521 19:51:00.858051 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 42.587135ms to process 1 records E0521 19:51:00.858064 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0521 19:51:00.858070 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 39.750387ms to process 0 records I0521 19:51:00.858077 1 tasks_processing.go:74] worker 11 stopped. W0521 19:51:00.860230 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:51:00.862888 1 gather_logs.go:145] no pods in namespace were found I0521 19:51:00.862905 1 tasks_processing.go:74] worker 45 stopped. I0521 19:51:00.862914 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 48.158574ms to process 0 records I0521 19:51:00.862996 1 tasks_processing.go:74] worker 60 stopped. I0521 19:51:00.863102 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=ac85a601fad7132343c10d00ac46298952250d599a465039a6f7c8af675d0290 I0521 19:51:00.863130 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=e32f83255ad6902d9ae1f80ad20995db3223fb5a2c611b34b95a61cce940d3d3 I0521 19:51:00.863140 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 46.154393ms to process 2 records I0521 19:51:00.863237 1 tasks_processing.go:74] worker 59 stopped. I0521 19:51:00.863405 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=159544372e5054a9f01ef3759360099b6695e9ba387b5109e16244a06759be8b I0521 19:51:00.863453 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=1bf40f7554c02766a77b4aaa54e3b0a640b721a1237086199f4899877d2b6fae I0521 19:51:00.863493 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=9e218d5fc21cce4667c3b2814b94ebb5bc4892cc7fa58fbc560cf1e9f0db17e3 I0521 19:51:00.863504 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 48.507494ms to process 3 records I0521 19:51:00.863627 1 recorder.go:75] Recording config/featuregate with fingerprint=5266ddc79f9952fd2a06a376cbf82f755e4e6eb0a3dcfd0676361786d40fed23 I0521 19:51:00.863639 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 45.728373ms to process 1 records I0521 19:51:00.863648 1 tasks_processing.go:74] worker 39 stopped. I0521 19:51:00.863660 1 tasks_processing.go:74] worker 30 stopped. I0521 19:51:00.863764 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=c86b8fc7f27a193f64a22f624a3009cff733e870e6a0e78bdd9d72a645bdfc77 I0521 19:51:00.863774 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0521 19:51:00.863783 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0521 19:51:00.863797 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=3bfbb170fed84ad4ce737ca9a6a24c0008d75c751f81b10fab7d9c8f06afd317 I0521 19:51:00.863821 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=1a1c44c22f0926424f298347696f01fc36ae98640f9d9347462f05c9d520cf31 I0521 19:51:00.863834 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 47.221812ms to process 3 records W0521 19:51:00.863985 1 operator.go:288] started I0521 19:51:00.864002 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0521 19:51:00.864041 1 tasks_processing.go:74] worker 42 stopped. I0521 19:51:00.864187 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=188d545c9268774441f00f0f4d87fa9e0119562b1ac292ce94115fe22076bf1d I0521 19:51:00.864340 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=6a3f1173f2460b5a8795d19c903775e5807267f576a133a8632f9022a7239462 I0521 19:51:00.864372 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=662a3b2b79d89fa9d809ebc47186173eb7aa9c19bdce10e0ee7171317899098c I0521 19:51:00.864414 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=07909fa67fbbbcf1bc49a04b3f34c578df1bfc3222ea93f95956b39d3a17d239 I0521 19:51:00.864453 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=4e146bb531bbf724deb6d0fbb506f01aac76fbf87712c00fd8bd12b6d611739b I0521 19:51:00.864501 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=52229dec5b721bb05a19e7d28f1159f04311086555f3b960432037d0684b76fe I0521 19:51:00.864539 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=437f646d1c2a9318278e2710e3bea97a717a0a0047f9022d221c9c17bc069eca I0521 19:51:00.864592 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=1a7bb48c6490577c233c5bbecbb6fe2610e4cd0e497eea0e43294276b9faae26 I0521 19:51:00.865178 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=ad074b5a1a85b7591733340f4ae065212d7b237658f32fea154178a482397c4f I0521 19:51:00.865235 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=29cf1cf56778fd238e463692d3f630e40ff432de4ca67fc30eec2ba84c437f33 I0521 19:51:00.865346 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=5be05f6c0bb1a62af43821d233c838459170e4ee301a38cfec1c2d382173b605 I0521 19:51:00.865362 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 47.34667ms to process 11 records I0521 19:51:00.867295 1 tasks_processing.go:74] worker 36 stopped. I0521 19:51:00.867308 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 50.883664ms to process 0 records I0521 19:51:00.869661 1 tasks_processing.go:74] worker 16 stopped. I0521 19:51:00.871230 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-jr6mj with fingerprint=34df12b4fab8eccb46589a3d70bf17e22829f4be469215bd6c8697a1da180912 I0521 19:51:00.871404 1 recorder.go:75] Recording config/pod/openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-1-122.ec2.internal with fingerprint=381426d9c3c5733f3b15e44a2213115bed357d2ec051a947d84f0ab8d51d76fe I0521 19:51:00.871526 1 recorder.go:75] Recording config/pod/openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-2-223.ec2.internal with fingerprint=7f53f70e830f8537b497d9801dab5eabd0cc8af9f022b892460d8b8e3b4d8dd2 I0521 19:51:00.871602 1 recorder.go:75] Recording config/running_containers with fingerprint=09ced6db50aa0ec87237c30777651876b68f83430570ee0829502ca2c9082545 I0521 19:51:00.871615 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 54.91529ms to process 4 records I0521 19:51:00.877651 1 tasks_processing.go:74] worker 35 stopped. I0521 19:51:00.877669 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 62.901041ms to process 0 records I0521 19:51:00.877764 1 tasks_processing.go:74] worker 62 stopped. I0521 19:51:00.877947 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=843b7ceeef2ade4f488719ee5bb04bce686159b38b9fe26f93b13bd043399ea2 I0521 19:51:00.878031 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=cfa2225014ed00cbc9a383e5560d6fcb45f695f0fbb6a0892e4d654823d8535b I0521 19:51:00.878040 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 63.217668ms to process 2 records I0521 19:51:00.878121 1 tasks_processing.go:74] worker 20 stopped. I0521 19:51:00.878483 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=51a9c748a907e7f8b5746b1080ac7935adf542579b00368a1fac270c3f20548d I0521 19:51:00.878674 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=58f23a31c05f00f75a4f5c18c335aacad3253b3fed2236d7b9fc4d8c59e75321 I0521 19:51:00.878682 1 gather.go:177] gatherer "clusterconfig" function "crds" took 60.599476ms to process 2 records I0521 19:51:00.879518 1 base_controller.go:82] Caches are synced for ConfigController I0521 19:51:00.879533 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0521 19:51:00.884168 1 controller.go:212] Source scaController *sca.Controller is not ready I0521 19:51:00.884179 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0521 19:51:00.884183 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:51:00.884186 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:51:00.884188 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0521 19:51:00.884212 1 controller.go:489] The operator is still being initialized I0521 19:51:00.884220 1 controller.go:512] The operator is healthy I0521 19:51:00.895819 1 prometheus_rules.go:88] Prometheus rules successfully created I0521 19:51:00.896194 1 tasks_processing.go:74] worker 21 stopped. I0521 19:51:00.896284 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 79.579639ms to process 0 records I0521 19:51:00.896455 1 tasks_processing.go:74] worker 47 stopped. I0521 19:51:00.896611 1 recorder.go:75] Recording config/olm_operators with fingerprint=afacc0190e559c4742051cbc5da2d24a2a595bb78ef60e8df9a2f3ee0d2f0149 I0521 19:51:00.896650 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 81.103008ms to process 1 records I0521 19:51:00.901886 1 tasks_processing.go:74] worker 19 stopped. I0521 19:51:00.901904 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 85.091853ms to process 0 records E0521 19:51:00.903488 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%271c66b408-74cc-49d9-a87e-c137f0b99dfa%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.10:60304->172.30.0.10:53: read: connection refused I0521 19:51:00.903500 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%271c66b408-74cc-49d9-a87e-c137f0b99dfa%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.10:60304->172.30.0.10:53: read: connection refused I0521 19:51:00.910117 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:51:00.918001 1 tasks_processing.go:74] worker 8 stopped. E0521 19:51:00.918025 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0521 19:51:00.918035 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0521 19:51:00.918039 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0521 19:51:00.918051 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0521 19:51:00.918087 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0521 19:51:00.918094 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0521 19:51:00.918098 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0521 19:51:00.918102 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0521 19:51:00.918140 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0521 19:51:00.918148 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0521 19:51:00.918153 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 103.233961ms to process 7 records I0521 19:51:00.931073 1 tasks_processing.go:74] worker 25 stopped. I0521 19:51:00.931381 1 recorder.go:75] Recording config/version with fingerprint=3e2464c593a4ff96e0b1f785c0b3d0203d4cd4b374fe49b2aa3259d62f4e9b92 I0521 19:51:00.931394 1 recorder.go:75] Recording config/id with fingerprint=4c351097d80185a0eed2c838acb5b9bc6c29beb2968dd44f999ba2c5a77bf106 I0521 19:51:00.931400 1 gather.go:177] gatherer "clusterconfig" function "version" took 116.561349ms to process 2 records I0521 19:51:00.940078 1 tasks_processing.go:74] worker 49 stopped. I0521 19:51:00.940094 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 123.094828ms to process 0 records I0521 19:51:00.946087 1 tasks_processing.go:74] worker 48 stopped. I0521 19:51:00.946116 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0521 19:51:00.946129 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 130.114869ms to process 1 records I0521 19:51:00.954378 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0521 19:51:00.957656 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.10:56084->172.30.0.10:53: read: connection refused I0521 19:51:00.957670 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.129.0.10:56084->172.30.0.10:53: read: connection refused I0521 19:51:00.964756 1 base_controller.go:82] Caches are synced for LoggingSyncer I0521 19:51:00.964770 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0521 19:51:00.998545 1 tasks_processing.go:74] worker 56 stopped. E0521 19:51:00.998564 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0521 19:51:00.998570 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2qegofej02cpr3mcmqunbo1ohbbjf8vt-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2qegofej02cpr3mcmqunbo1ohbbjf8vt-primary-cert-bundle-secret" not found I0521 19:51:00.998629 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=e0fea416bb9459aaac335d44f3b6ae3efab19eee6e33ae985475a4a40e2c02e5 I0521 19:51:00.998641 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 182.150929ms to process 1 records W0521 19:51:01.858870 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:51:02.072862 1 gather_cluster_operator_pods_and_events.go:121] Found 20 pods with 24 containers I0521 19:51:02.072877 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1048576 bytes I0521 19:51:02.073380 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-lkzvj pod in namespace openshift-dns (previous: false). I0521 19:51:02.291776 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0521 19:51:02.296798 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-lkzvj pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-lkzvj\" is waiting to start: ContainerCreating" I0521 19:51:02.296814 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-lkzvj\" is waiting to start: ContainerCreating" I0521 19:51:02.296823 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-lkzvj pod in namespace openshift-dns (previous: false). I0521 19:51:02.476106 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-lkzvj pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-lkzvj\" is waiting to start: ContainerCreating" I0521 19:51:02.476123 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-lkzvj\" is waiting to start: ContainerCreating" I0521 19:51:02.476156 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-mmnrf pod in namespace openshift-dns (previous: false). I0521 19:51:02.702693 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-mmnrf pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-mmnrf\" is waiting to start: ContainerCreating" I0521 19:51:02.702714 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-mmnrf\" is waiting to start: ContainerCreating" I0521 19:51:02.702723 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-mmnrf pod in namespace openshift-dns (previous: false). W0521 19:51:02.858982 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:51:02.878234 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-mmnrf pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-mmnrf\" is waiting to start: ContainerCreating" I0521 19:51:02.878267 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-mmnrf\" is waiting to start: ContainerCreating" I0521 19:51:02.878299 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-sngms pod in namespace openshift-dns (previous: false). I0521 19:51:02.896161 1 tasks_processing.go:74] worker 29 stopped. I0521 19:51:02.896215 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=77b55329b95825791d5c0262dbb6a8ebf05c65523dfb3d34e6a7584cb128110e I0521 19:51:02.896277 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=55cbfc5c896da4bf2dd81408549a1a2a980b6403f4007afd2e19aec2a7edeef9 I0521 19:51:02.896327 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0521 19:51:02.896353 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=66319ca6cfcba794d20bd14fb6e745339004dfbafd4237494956d0e92057cf93 I0521 19:51:02.896371 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0521 19:51:02.896392 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=6a1321adc30a94f57206b0510168810ab988432641042dd121cbaaa9a7901142 I0521 19:51:02.896424 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=0f3082875c1ce9a5c7bd6bed6a146cba53bfc5574d1ae1983482c552733c94d3 I0521 19:51:02.896448 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=321c805498921d67b9bc5ed53d91f2e304fe8c6ec722ef7288ce09cc41f46bf1 I0521 19:51:02.896459 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=d1feb5855e24fe0975c4644635da2365b35fb358d8cb135af4651d46d60766b6 I0521 19:51:02.896478 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=90189fa5d8d04219898823b9d5e0e8a8683064644c30d51ddb9037b7273ae319 I0521 19:51:02.896488 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0521 19:51:02.896503 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=6284eab2db51f6c42889c73912f7043b95488126568719233b24606368ddbab7 I0521 19:51:02.896514 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0521 19:51:02.896529 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=9c00291c43097b97f0ca97e686cfbdff242fd24428585a096a8e2f12dacb4d58 I0521 19:51:02.896539 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0521 19:51:02.896561 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=de97c50f95ffa9e4e9ca93bfaf2be8ac023eda71036b6a3997d12c073a9f0f6c I0521 19:51:02.896571 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0521 19:51:02.896586 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=7102a8adfe7cf8dad1c2f6c5feac3e63bda59e6d10a8c869d54eb0706fad1874 I0521 19:51:02.896743 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=2770e12b91efebe7a519804d93e79075e6234361bef917fad663b5c0b3f71579 I0521 19:51:02.896754 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0521 19:51:02.896760 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0521 19:51:02.896780 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0521 19:51:02.896821 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=4f9b599f842b7823818a44fcd87901ada920abb508bfc5f2b8306db4987dfe6d I0521 19:51:02.896846 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=895bd3e70c18852ac70bb8c32412918775a402910c670a10c0917c956d9ae2da I0521 19:51:02.896856 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0521 19:51:02.896871 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=3b0bea6e25a5d79c2701e47853a14b1f4d245c15aa0a5bfe3c46be65b3c3f07b I0521 19:51:02.896880 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0521 19:51:02.896893 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=2f6c57ed3f8d5568fd32e969ed0dcf9feea20fa36296c0a7a4c7e3fff19b5fc8 I0521 19:51:02.896908 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=19eb0b4ac74b093296445310c404e1e46ee9ec0f01f9d28a5a3aec85bbaafd9d I0521 19:51:02.896930 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=b505c06702b052a25668354118ee315118d1e3476447b8ca69bfd443d4f10716 I0521 19:51:02.896946 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=283fb760ecbf49a500e98954918ac3665a30bcbad5b4ebfa5edab72809d91eca I0521 19:51:02.896967 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=126b583aee378841b952c1da813728b19813a8d7e329ed7616a6bcdc0e79cbd4 I0521 19:51:02.896976 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0521 19:51:02.897003 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=741d196aa12cc3e609dab199862ac0b3c9e8c622721b87896d1897d43a965dbe I0521 19:51:02.897020 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0521 19:51:02.897028 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0521 19:51:02.897034 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.081408222s to process 36 records I0521 19:51:03.106570 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-sngms pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-sngms\" is waiting to start: ContainerCreating" I0521 19:51:03.106589 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-sngms\" is waiting to start: ContainerCreating" I0521 19:51:03.106600 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-sngms pod in namespace openshift-dns (previous: false). I0521 19:51:03.280436 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-sngms pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-sngms\" is waiting to start: ContainerCreating" I0521 19:51:03.280455 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-sngms\" is waiting to start: ContainerCreating" I0521 19:51:03.280465 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-968bx pod in namespace openshift-dns (previous: false). I0521 19:51:03.475574 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:51:03.475592 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-jtx2s pod in namespace openshift-dns (previous: false). I0521 19:51:03.677721 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:51:03.677740 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-l8fmq pod in namespace openshift-dns (previous: false). W0521 19:51:03.859710 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:51:03.878271 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:51:03.878356 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6bbcbf5849-gsvkh pod in namespace openshift-image-registry (previous: false). I0521 19:51:04.075701 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-6bbcbf5849-gsvkh pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-6bbcbf5849-gsvkh\" is waiting to start: ContainerCreating" I0521 19:51:04.075724 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-6bbcbf5849-gsvkh\" is waiting to start: ContainerCreating" I0521 19:51:04.075760 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-8477f45fd6-jpdhf pod in namespace openshift-image-registry (previous: false). I0521 19:51:04.277882 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-8477f45fd6-jpdhf pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-8477f45fd6-jpdhf\" is waiting to start: ContainerCreating" I0521 19:51:04.277931 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-8477f45fd6-jpdhf\" is waiting to start: ContainerCreating" I0521 19:51:04.277965 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-8477f45fd6-zljnv pod in namespace openshift-image-registry (previous: false). I0521 19:51:04.511333 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-8477f45fd6-zljnv pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-8477f45fd6-zljnv\" is waiting to start: ContainerCreating" I0521 19:51:04.511353 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-8477f45fd6-zljnv\" is waiting to start: ContainerCreating" I0521 19:51:04.511364 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-9vcd6 pod in namespace openshift-image-registry (previous: false). I0521 19:51:04.689332 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:51:04.689351 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-dptpz pod in namespace openshift-image-registry (previous: false). W0521 19:51:04.859167 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:51:04.876441 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:51:04.876458 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-mvqfr pod in namespace openshift-image-registry (previous: false). I0521 19:51:05.079932 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:51:05.079952 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5995584fd8-7qtd9 pod in namespace openshift-ingress (previous: false). I0521 19:51:05.280934 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5995584fd8-7qtd9 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5995584fd8-7qtd9\" is waiting to start: ContainerCreating" I0521 19:51:05.280951 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5995584fd8-7qtd9\" is waiting to start: ContainerCreating" I0521 19:51:05.280962 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5995584fd8-7zk2f pod in namespace openshift-ingress (previous: false). I0521 19:51:05.501162 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5995584fd8-7zk2f pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5995584fd8-7zk2f\" is waiting to start: ContainerCreating" I0521 19:51:05.501179 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5995584fd8-7zk2f\" is waiting to start: ContainerCreating" I0521 19:51:05.501190 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-d5464bc65-zwpnd pod in namespace openshift-ingress (previous: false). I0521 19:51:05.681986 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-d5464bc65-zwpnd pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-d5464bc65-zwpnd\" is waiting to start: ContainerCreating" I0521 19:51:05.682004 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-d5464bc65-zwpnd\" is waiting to start: ContainerCreating" I0521 19:51:05.682034 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-5v7xs pod in namespace openshift-ingress-canary (previous: false). W0521 19:51:05.859095 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0521 19:51:05.859121 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:51:05.859137 1 tasks_processing.go:74] worker 43 stopped. E0521 19:51:05.859147 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:51:05.859157 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0521 19:51:05.859170 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0521 19:51:05.859183 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.042250843s to process 1 records I0521 19:51:05.879668 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-5v7xs pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-5v7xs\" is waiting to start: ContainerCreating" I0521 19:51:05.879683 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-5v7xs\" is waiting to start: ContainerCreating" I0521 19:51:05.879707 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-n49fb pod in namespace openshift-ingress-canary (previous: false). I0521 19:51:06.075278 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-n49fb pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-n49fb\" is waiting to start: ContainerCreating" I0521 19:51:06.075298 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-n49fb\" is waiting to start: ContainerCreating" I0521 19:51:06.075327 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-nplnm pod in namespace openshift-ingress-canary (previous: false). I0521 19:51:06.278434 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-nplnm pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-nplnm\" is waiting to start: ContainerCreating" I0521 19:51:06.278451 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-nplnm\" is waiting to start: ContainerCreating" I0521 19:51:06.278463 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for migrator container migrator-7d5f866c57-vvkqj pod in namespace openshift-kube-storage-version-migrator (previous: false). I0521 19:51:06.478392 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for graceful-termination container migrator-7d5f866c57-vvkqj pod in namespace openshift-kube-storage-version-migrator (previous: false). I0521 19:51:06.677470 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-storage-version-migrator-operator container kube-storage-version-migrator-operator-74848b4cb9-lvvff pod in namespace openshift-kube-storage-version-migrator-operator (previous: false). I0521 19:51:06.890175 1 tasks_processing.go:74] worker 38 stopped. I0521 19:51:06.890334 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=5c305ffa599635b44ccf5604386fddeaf9cc0aeb8f835889643f2e58e8f6ce72 I0521 19:51:06.890459 1 recorder.go:75] Recording events/openshift-dns with fingerprint=b81aa919a651a1ff77b56a232ba96f372847967ee11d722fb02684211f940f3e I0521 19:51:06.890668 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=5b423673c679ab4c34fb616912d8f977ed5965d0abba2c1941d1fd1d2ab6cd67 I0521 19:51:06.890735 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=20fff95e8865f9335da78ed973c70413d78d42dd6ab32d1e9f45316f374d8286 I0521 19:51:06.890837 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=9733bc819c650bd6464d74b90963c1edf7b22e241b25f606fc97aa6b13285fc6 I0521 19:51:06.890881 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=cf2b7d498a8d9fc404775c97c0990acd3591f08244ac4f1336b6ae52f2f58299 I0521 19:51:06.890920 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator with fingerprint=f0ebebdc2b0cfc5bb3335497073ff144dd31f6ef6191a8f196b144eda8ce0321 I0521 19:51:06.891017 1 recorder.go:75] Recording events/openshift-kube-storage-version-migrator-operator with fingerprint=804e45fa473ab8a84a6e2b9406318cfb9f91c81f8d9462ba8aca7cd14a001c5c I0521 19:51:06.891231 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-lkzvj with fingerprint=051332f2edb821ddf192004c39d0458f351e62929325f51323c9236490797391 I0521 19:51:06.891721 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-mmnrf with fingerprint=ba6224fe5ec5e26e1ed5e69d20a30c1bc504556158d245becaa145dc25c6ea31 I0521 19:51:06.891873 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-sngms with fingerprint=7b4e702c2fd2546c7a064e0b6f2142bbee56f1e312746341b334834705561c4a I0521 19:51:06.892052 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-6bbcbf5849-gsvkh with fingerprint=4340439ea4003f34200a748a761d3b02a33bb39eb6a09020c7d2a3ffe853f929 I0521 19:51:06.892264 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-8477f45fd6-jpdhf with fingerprint=385d9123b1bb0369ab12c38e0fc7a547f3d5f4d2797c59788e2b7bbc1ba825a4 I0521 19:51:06.892370 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-8477f45fd6-zljnv with fingerprint=f9f50e12e41e575200b3c5c4d4dcde031c345e5de8052acd12f6076999c0846b I0521 19:51:06.892430 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-5v7xs with fingerprint=20e3ef990fe4e144f5f19597fbb77ac5642c925161004bce226beb6a81903df9 I0521 19:51:06.892484 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-n49fb with fingerprint=3dd5aa84770229fb441bea92e8ac79e573af8718182069c624af947176838c25 I0521 19:51:06.892537 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-nplnm with fingerprint=d6d5597b145e9747ddd9db8a4842a4669a4b77f764c78e3a80ad902a778ff16f I0521 19:51:06.892552 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-vvkqj/migrator_current.log with fingerprint=b4db00a51928a7cf664f83a7b0d958c58d97be7f91de67fab1582b798de6b205 I0521 19:51:06.892557 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator/logs/migrator-7d5f866c57-vvkqj/graceful-termination_current.log with fingerprint=94d0dd606fb3d7da6933d96f7f024dd4bb0aa58f4db57bdfff53d53f1bcc3990 I0521 19:51:06.892624 1 recorder.go:75] Recording config/pod/openshift-kube-storage-version-migrator-operator/logs/kube-storage-version-migrator-operator-74848b4cb9-lvvff/kube-storage-version-migrator-operator_current.log with fingerprint=705b5d63b9e5324250eaf1b95a14962c6d6527eef248539de3b7133742812e88 I0521 19:51:06.892633 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 6.075496899s to process 20 records I0521 19:51:11.867447 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:51:13.670009 1 tasks_processing.go:74] worker 15 stopped. I0521 19:51:13.670049 1 recorder.go:75] Recording config/installplans with fingerprint=95dfd5f33a9a46199239158a0ec0183b629818eca24ae41f4d706d7a3d604aec I0521 19:51:13.670084 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.853222012s to process 1 records I0521 19:51:13.811117 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:51:14.013165 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:51:14.424697 1 tasks_processing.go:74] worker 44 stopped. I0521 19:51:14.424955 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=c5e0ad81555672a1ae6992800f0e8a9ad0a7406d6b5e671cb8278224bb66f9c4 I0521 19:51:14.424984 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.607351623s to process 1 records E0521 19:51:14.425041 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.61s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0521 19:51:14.426145 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0521 19:51:14.426161 1 periodic.go:209] Running workloads gatherer I0521 19:51:14.426179 1 tasks_processing.go:45] number of workers: 2 I0521 19:51:14.426191 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:51:14.426196 1 tasks_processing.go:71] worker 1 working on workload_info task. I0521 19:51:14.426277 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:51:14.426372 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0521 19:51:14.454623 1 tasks_processing.go:74] worker 0 stopped. I0521 19:51:14.454641 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 28.228386ms to process 0 records I0521 19:51:14.526940 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 22s for image data I0521 19:51:14.535444 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (9ms) I0521 19:51:14.544548 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (9ms) I0521 19:51:14.553394 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (9ms) I0521 19:51:14.561476 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (8ms) I0521 19:51:14.569485 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (8ms) I0521 19:51:14.577738 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (8ms) I0521 19:51:14.585299 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (8ms) I0521 19:51:14.592914 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (8ms) I0521 19:51:14.600734 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (8ms) I0521 19:51:14.610272 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (10ms) I0521 19:51:14.636520 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (26ms) I0521 19:51:14.735849 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (99ms) I0521 19:51:14.835699 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (100ms) I0521 19:51:14.942624 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (107ms) I0521 19:51:15.036057 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (93ms) I0521 19:51:15.136562 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (100ms) I0521 19:51:15.244148 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (108ms) I0521 19:51:15.335542 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (91ms) I0521 19:51:15.435717 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0521 19:51:15.536999 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (101ms) I0521 19:51:15.644199 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (107ms) I0521 19:51:15.736039 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (92ms) I0521 19:51:15.836312 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (100ms) I0521 19:51:15.936101 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (100ms) I0521 19:51:16.036007 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0521 19:51:16.136389 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0521 19:51:16.236118 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (100ms) I0521 19:51:16.340180 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (104ms) I0521 19:51:16.440054 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0521 19:51:16.535977 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (96ms) I0521 19:51:16.636464 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0521 19:51:16.736225 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (100ms) I0521 19:51:16.835811 1 gather_workloads_info.go:387] No image sha256:f68c0da8268f23078f5f760a91a3c8212f2195acf18c231f6e034207a67adeec (100ms) I0521 19:51:16.835848 1 tasks_processing.go:74] worker 1 stopped. E0521 19:51:16.835859 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0521 19:51:16.836169 1 recorder.go:75] Recording config/workload_info with fingerprint=bd4f6bfaace2d06287dde4b159508755c88c25125e865c5869d69c46d6a33634 I0521 19:51:16.836188 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.409641934s to process 1 records E0521 19:51:16.836213 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.41s with: function \"workload_info\" failed with an error" I0521 19:51:16.837284 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0521 19:51:16.837300 1 periodic.go:209] Running conditional gatherer I0521 19:51:16.843074 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0521 19:51:16.849462 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.129.0.10:33207->172.30.0.10:53: read: connection refused E0521 19:51:16.849711 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0521 19:51:16.849769 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0521 19:51:16.855804 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0521 19:51:16.855818 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:51:16.855823 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:51:16.855827 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:51:16.855830 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:51:16.855834 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:51:16.855837 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:51:16.855840 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:51:16.855843 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:51:16.855846 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0521 19:51:16.855870 1 tasks_processing.go:45] number of workers: 3 I0521 19:51:16.855882 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:51:16.855886 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0521 19:51:16.855895 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:51:16.855914 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0521 19:51:16.855914 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:51:16.855920 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0521 19:51:16.855924 1 tasks_processing.go:74] worker 1 stopped. I0521 19:51:16.855977 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0521 19:51:16.855991 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 914ns to process 1 records I0521 19:51:16.856029 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0521 19:51:16.856039 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.299µs to process 1 records I0521 19:51:16.856044 1 tasks_processing.go:74] worker 0 stopped. I0521 19:51:16.856169 1 tasks_processing.go:74] worker 2 stopped. I0521 19:51:16.856181 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 239.515µs to process 0 records I0521 19:51:16.856201 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.129.0.10:33207->172.30.0.10:53: read: connection refused I0521 19:51:16.856218 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0521 19:51:16.877457 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=e9135c28e56cb75f8da1d286dffa953cf04b18f00a96071437dfa62290d775bb I0521 19:51:16.877635 1 diskrecorder.go:70] Writing 118 records to /var/lib/insights-operator/insights-2026-05-21-195116.tar.gz I0521 19:51:16.885755 1 diskrecorder.go:51] Wrote 118 records to disk in 8ms I0521 19:51:16.885794 1 periodic.go:278] Gathering cluster info every 2h0m0s I0521 19:51:16.885811 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0521 19:51:22.359191 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:52:24.494988 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="4c83895b7016fc9a76a14324d49305e4935046628355724cc3f533068151b684") W0521 19:52:24.495021 1 builder.go:160] Restart triggered because of file /var/run/configmaps/service-ca-bundle/service-ca.crt was created I0521 19:52:24.495308 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="437ca06fe575ad94d9e04075a568d015f52909bbf4899d0ec25f26c12433409a") I0521 19:52:24.495333 1 base_controller.go:181] Shutting down LoggingSyncer ... I0521 19:52:24.495373 1 base_controller.go:123] Shutting down worker of ConfigController controller ...