W0504 20:39:43.378059 1 cmd.go:257] Using insecure, self-signed certificates I0504 20:39:43.669613 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:39:43.669915 1 observer_polling.go:159] Starting file observer I0504 20:39:44.031200 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0504 20:39:44.031387 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0504 20:39:44.031945 1 secure_serving.go:57] Forcing use of http/1.1 only W0504 20:39:44.031966 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0504 20:39:44.031970 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0504 20:39:44.031975 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0504 20:39:44.031979 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0504 20:39:44.031981 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0504 20:39:44.031984 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0504 20:39:44.032363 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0504 20:39:44.037045 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0504 20:39:44.037101 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"ba5c066e-2163-4573-9d86-b503974dbc3f", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0504 20:39:44.038922 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0504 20:39:44.038937 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0504 20:39:44.038971 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0504 20:39:44.038984 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0504 20:39:44.038998 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0504 20:39:44.039042 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0504 20:39:44.039157 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-1184084844/tls.crt::/tmp/serving-cert-1184084844/tls.key" I0504 20:39:44.039421 1 secure_serving.go:213] Serving securely on [::]:8443 I0504 20:39:44.039447 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0504 20:39:44.044176 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0504 20:39:44.044198 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0504 20:39:44.044292 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0504 20:39:44.052343 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 20:39:44.052362 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0504 20:39:44.057756 1 secretconfigobserver.go:119] support secret does not exist I0504 20:39:44.063332 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0504 20:39:44.068813 1 secretconfigobserver.go:119] support secret does not exist I0504 20:39:44.072056 1 recorder.go:176] Pruning old reports every 4h50m15s, max age is 288h0m0s I0504 20:39:44.079272 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0504 20:39:44.079359 1 insightsreport.go:296] Starting report retriever I0504 20:39:44.079441 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0504 20:39:44.079275 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0504 20:39:44.079285 1 periodic.go:216] Running clusterconfig gatherer I0504 20:39:44.079976 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0504 20:39:44.080289 1 tasks_processing.go:45] number of workers: 64 I0504 20:39:44.080330 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 20:39:44.080336 1 tasks_processing.go:69] worker 8 listening for tasks. I0504 20:39:44.080350 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 20:39:44.080347 1 tasks_processing.go:69] worker 4 listening for tasks. I0504 20:39:44.080355 1 tasks_processing.go:69] worker 2 listening for tasks. I0504 20:39:44.080363 1 tasks_processing.go:69] worker 7 listening for tasks. I0504 20:39:44.080352 1 tasks_processing.go:69] worker 5 listening for tasks. I0504 20:39:44.080370 1 tasks_processing.go:69] worker 3 listening for tasks. I0504 20:39:44.080366 1 tasks_processing.go:69] worker 33 listening for tasks. I0504 20:39:44.080377 1 tasks_processing.go:69] worker 9 listening for tasks. I0504 20:39:44.080378 1 tasks_processing.go:71] worker 3 working on service_accounts task. I0504 20:39:44.080381 1 tasks_processing.go:69] worker 57 listening for tasks. I0504 20:39:44.080381 1 tasks_processing.go:71] worker 2 working on storage_classes task. I0504 20:39:44.080386 1 tasks_processing.go:69] worker 15 listening for tasks. I0504 20:39:44.080387 1 tasks_processing.go:69] worker 58 listening for tasks. I0504 20:39:44.080390 1 tasks_processing.go:69] worker 51 listening for tasks. I0504 20:39:44.080393 1 tasks_processing.go:71] worker 15 working on ingress task. I0504 20:39:44.080359 1 tasks_processing.go:71] worker 0 working on networks task. I0504 20:39:44.080403 1 tasks_processing.go:69] worker 52 listening for tasks. I0504 20:39:44.080398 1 tasks_processing.go:69] worker 14 listening for tasks. I0504 20:39:44.080413 1 tasks_processing.go:69] worker 63 listening for tasks. I0504 20:39:44.080416 1 tasks_processing.go:69] worker 23 listening for tasks. I0504 20:39:44.080405 1 tasks_processing.go:69] worker 60 listening for tasks. I0504 20:39:44.080424 1 tasks_processing.go:71] worker 1 working on feature_gates task. I0504 20:39:44.080426 1 tasks_processing.go:69] worker 35 listening for tasks. I0504 20:39:44.080430 1 tasks_processing.go:69] worker 13 listening for tasks. I0504 20:39:44.080427 1 tasks_processing.go:69] worker 62 listening for tasks. I0504 20:39:44.080357 1 tasks_processing.go:71] worker 8 working on openshift_machine_api_events task. I0504 20:39:44.080438 1 tasks_processing.go:69] worker 56 listening for tasks. I0504 20:39:44.080437 1 tasks_processing.go:69] worker 61 listening for tasks. I0504 20:39:44.080372 1 tasks_processing.go:69] worker 20 listening for tasks. I0504 20:39:44.080368 1 tasks_processing.go:69] worker 34 listening for tasks. I0504 20:39:44.080453 1 tasks_processing.go:71] worker 9 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0504 20:39:44.080455 1 tasks_processing.go:69] worker 17 listening for tasks. I0504 20:39:44.080373 1 tasks_processing.go:71] worker 5 working on cluster_apiserver task. I0504 20:39:44.080465 1 tasks_processing.go:69] worker 45 listening for tasks. I0504 20:39:44.080459 1 tasks_processing.go:69] worker 29 listening for tasks. I0504 20:39:44.080472 1 tasks_processing.go:69] worker 42 listening for tasks. I0504 20:39:44.080478 1 tasks_processing.go:69] worker 38 listening for tasks. I0504 20:39:44.080388 1 tasks_processing.go:71] worker 57 working on aggregated_monitoring_cr_names task. I0504 20:39:44.080485 1 tasks_processing.go:69] worker 26 listening for tasks. I0504 20:39:44.080489 1 tasks_processing.go:71] worker 38 working on image_pruners task. I0504 20:39:44.080492 1 tasks_processing.go:69] worker 44 listening for tasks. I0504 20:39:44.080495 1 tasks_processing.go:71] worker 26 working on active_alerts task. I0504 20:39:44.080496 1 tasks_processing.go:69] worker 40 listening for tasks. I0504 20:39:44.080502 1 tasks_processing.go:69] worker 37 listening for tasks. I0504 20:39:44.080509 1 tasks_processing.go:69] worker 41 listening for tasks. I0504 20:39:44.080511 1 tasks_processing.go:69] worker 28 listening for tasks. I0504 20:39:44.080489 1 tasks_processing.go:69] worker 39 listening for tasks. I0504 20:39:44.080509 1 tasks_processing.go:69] worker 36 listening for tasks. I0504 20:39:44.080398 1 tasks_processing.go:69] worker 59 listening for tasks. I0504 20:39:44.080521 1 tasks_processing.go:69] worker 27 listening for tasks. W0504 20:39:44.080522 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:39:44.080405 1 tasks_processing.go:69] worker 22 listening for tasks. I0504 20:39:44.080531 1 tasks_processing.go:69] worker 47 listening for tasks. I0504 20:39:44.080414 1 tasks_processing.go:69] worker 53 listening for tasks. I0504 20:39:44.080421 1 tasks_processing.go:69] worker 54 listening for tasks. I0504 20:39:44.080423 1 tasks_processing.go:69] worker 10 listening for tasks. I0504 20:39:44.080539 1 tasks_processing.go:69] worker 46 listening for tasks. I0504 20:39:44.080429 1 tasks_processing.go:69] worker 55 listening for tasks. I0504 20:39:44.080434 1 tasks_processing.go:69] worker 24 listening for tasks. I0504 20:39:44.080551 1 tasks_processing.go:71] worker 55 working on validating_webhook_configurations task. I0504 20:39:44.080554 1 tasks_processing.go:71] worker 24 working on schedulers task. I0504 20:39:44.080357 1 tasks_processing.go:69] worker 6 listening for tasks. I0504 20:39:44.080442 1 tasks_processing.go:69] worker 30 listening for tasks. I0504 20:39:44.080448 1 tasks_processing.go:69] worker 16 listening for tasks. I0504 20:39:44.080448 1 tasks_processing.go:69] worker 31 listening for tasks. I0504 20:39:44.080455 1 tasks_processing.go:69] worker 32 listening for tasks. I0504 20:39:44.080470 1 tasks_processing.go:69] worker 11 listening for tasks. I0504 20:39:44.080476 1 tasks_processing.go:69] worker 25 listening for tasks. I0504 20:39:44.080477 1 tasks_processing.go:69] worker 19 listening for tasks. I0504 20:39:44.080484 1 tasks_processing.go:69] worker 43 listening for tasks. I0504 20:39:44.080396 1 tasks_processing.go:69] worker 21 listening for tasks. I0504 20:39:44.080538 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 30.582µs to process 0 records I0504 20:39:44.080379 1 tasks_processing.go:69] worker 50 listening for tasks. I0504 20:39:44.080631 1 tasks_processing.go:71] worker 34 working on openstack_dataplanenodesets task. I0504 20:39:44.080639 1 tasks_processing.go:71] worker 56 working on operators task. I0504 20:39:44.080379 1 tasks_processing.go:69] worker 49 listening for tasks. I0504 20:39:44.080671 1 tasks_processing.go:71] worker 13 working on image_registries task. I0504 20:39:44.080677 1 tasks_processing.go:71] worker 45 working on nodenetworkstates task. I0504 20:39:44.080695 1 tasks_processing.go:71] worker 29 working on node_features task. I0504 20:39:44.080705 1 tasks_processing.go:71] worker 17 working on metrics task. W0504 20:39:44.080738 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:39:44.080790 1 tasks_processing.go:71] worker 50 working on certificate_signing_requests task. I0504 20:39:44.080794 1 tasks_processing.go:69] worker 48 listening for tasks. I0504 20:39:44.080805 1 tasks_processing.go:71] worker 48 working on support_secret task. I0504 20:39:44.080807 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 78.135µs to process 0 records I0504 20:39:44.080381 1 tasks_processing.go:71] worker 33 working on container_images task. I0504 20:39:44.080829 1 tasks_processing.go:71] worker 23 working on image task. I0504 20:39:44.080376 1 tasks_processing.go:71] worker 4 working on operators_pods_and_events task. I0504 20:39:44.080856 1 tasks_processing.go:71] worker 63 working on sap_datahubs task. I0504 20:39:44.080873 1 tasks_processing.go:71] worker 59 working on infrastructures task. I0504 20:39:44.080899 1 tasks_processing.go:71] worker 41 working on overlapping_namespace_uids task. I0504 20:39:44.080746 1 tasks_processing.go:71] worker 49 working on clusterroles task. I0504 20:39:44.080937 1 tasks_processing.go:71] worker 47 working on pod_network_connectivity_checks task. I0504 20:39:44.080857 1 tasks_processing.go:71] worker 42 working on version task. I0504 20:39:44.080968 1 tasks_processing.go:71] worker 39 working on authentication task. I0504 20:39:44.081010 1 tasks_processing.go:71] worker 27 working on sap_config task. I0504 20:39:44.081021 1 tasks_processing.go:71] worker 37 working on ceph_cluster task. I0504 20:39:44.080976 1 tasks_processing.go:71] worker 53 working on machines task. I0504 20:39:44.080441 1 tasks_processing.go:69] worker 18 listening for tasks. I0504 20:39:44.081241 1 tasks_processing.go:71] worker 18 working on node_logs task. I0504 20:39:44.080755 1 tasks_processing.go:71] worker 30 working on install_plans task. I0504 20:39:44.080751 1 tasks_processing.go:71] worker 6 working on cost_management_metrics_configs task. I0504 20:39:44.081012 1 tasks_processing.go:71] worker 22 working on oauths task. I0504 20:39:44.080759 1 tasks_processing.go:71] worker 16 working on jaegers task. I0504 20:39:44.080767 1 tasks_processing.go:71] worker 32 working on openshift_logging task. I0504 20:39:44.081436 1 tasks_processing.go:71] worker 60 working on silenced_alerts task. I0504 20:39:44.080763 1 tasks_processing.go:71] worker 31 working on machine_autoscalers task. W0504 20:39:44.081472 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:39:44.080770 1 tasks_processing.go:71] worker 11 working on openstack_version task. I0504 20:39:44.081487 1 tasks_processing.go:71] worker 60 working on openstack_dataplanedeployments task. I0504 20:39:44.080768 1 tasks_processing.go:71] worker 14 working on qemu_kubevirt_launcher_logs task. I0504 20:39:44.081478 1 tasks_processing.go:69] worker 12 listening for tasks. I0504 20:39:44.081504 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 34.544µs to process 0 records I0504 20:39:44.080371 1 tasks_processing.go:71] worker 7 working on proxies task. I0504 20:39:44.080774 1 tasks_processing.go:71] worker 25 working on machine_config_pools task. I0504 20:39:44.081615 1 tasks_processing.go:74] worker 12 stopped. I0504 20:39:44.080778 1 tasks_processing.go:71] worker 52 working on container_runtime_configs task. I0504 20:39:44.080772 1 tasks_processing.go:71] worker 58 working on mutating_webhook_configurations task. I0504 20:39:44.080778 1 tasks_processing.go:71] worker 19 working on dvo_metrics task. I0504 20:39:44.080782 1 tasks_processing.go:71] worker 61 working on ingress_certificates task. I0504 20:39:44.080782 1 tasks_processing.go:71] worker 43 working on config_maps task. I0504 20:39:44.080636 1 tasks_processing.go:71] worker 62 working on nodenetworkconfigurationpolicies task. I0504 20:39:44.080787 1 tasks_processing.go:71] worker 20 working on machine_sets task. I0504 20:39:44.080775 1 tasks_processing.go:71] worker 51 working on pdbs task. I0504 20:39:44.080786 1 tasks_processing.go:71] worker 21 working on machine_configs task. I0504 20:39:44.080796 1 tasks_processing.go:71] worker 17 working on openstack_controlplanes task. I0504 20:39:44.080867 1 tasks_processing.go:71] worker 35 working on storage_cluster task. I0504 20:39:44.080989 1 tasks_processing.go:71] worker 46 working on tsdb_status task. I0504 20:39:44.080995 1 tasks_processing.go:71] worker 28 working on crds task. W0504 20:39:44.082070 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:39:44.082095 1 tasks_processing.go:74] worker 46 stopped. I0504 20:39:44.082107 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 46.613µs to process 0 records I0504 20:39:44.080997 1 tasks_processing.go:71] worker 36 working on machine_healthchecks task. I0504 20:39:44.081004 1 tasks_processing.go:71] worker 26 working on lokistack task. I0504 20:39:44.081006 1 tasks_processing.go:71] worker 44 working on monitoring_persistent_volumes task. I0504 20:39:44.080982 1 tasks_processing.go:71] worker 54 working on olm_operators task. I0504 20:39:44.080986 1 tasks_processing.go:71] worker 10 working on nodes task. I0504 20:39:44.081002 1 tasks_processing.go:71] worker 40 working on sap_pods task. I0504 20:39:44.084985 1 tasks_processing.go:74] worker 45 stopped. I0504 20:39:44.085000 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 4.292903ms to process 0 records I0504 20:39:44.086907 1 tasks_processing.go:74] worker 6 stopped. I0504 20:39:44.086921 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 5.610268ms to process 0 records I0504 20:39:44.089849 1 tasks_processing.go:74] worker 53 stopped. E0504 20:39:44.089860 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0504 20:39:44.089868 1 gather.go:177] gatherer "clusterconfig" function "machines" took 8.677804ms to process 0 records I0504 20:39:44.089982 1 tasks_processing.go:74] worker 1 stopped. I0504 20:39:44.090118 1 recorder.go:75] Recording config/featuregate with fingerprint=758e72ee72aa2e056bcabbccfd63b35f901629af178ff5bc9cf5ed46b1d68dcc I0504 20:39:44.090149 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 9.543853ms to process 1 records I0504 20:39:44.090166 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 8.542179ms to process 0 records I0504 20:39:44.090177 1 tasks_processing.go:74] worker 32 stopped. I0504 20:39:44.090225 1 tasks_processing.go:74] worker 16 stopped. I0504 20:39:44.090238 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 8.801219ms to process 0 records E0504 20:39:44.090511 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0504 20:39:44.090523 1 tasks_processing.go:74] worker 29 stopped. I0504 20:39:44.090536 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 9.813002ms to process 0 records I0504 20:39:44.090635 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0504 20:39:44.090649 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0504 20:39:44.090654 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0504 20:39:44.090658 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0504 20:39:44.090670 1 controller.go:531] The operator is still being initialized I0504 20:39:44.090676 1 controller.go:554] The operator is healthy I0504 20:39:44.091592 1 tasks_processing.go:74] worker 5 stopped. I0504 20:39:44.091735 1 recorder.go:75] Recording config/apiserver with fingerprint=6c9f445a1a2ee677092a611138700c0d85c44dd887cc511feffed344843bf9c6 I0504 20:39:44.091748 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 11.119633ms to process 1 records I0504 20:39:44.092114 1 tasks_processing.go:74] worker 15 stopped. I0504 20:39:44.092268 1 recorder.go:75] Recording config/ingress with fingerprint=7c5913397bf564043e4b90ac79857b342005cecfded7201a833f9aaf42fb4d93 I0504 20:39:44.092282 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 11.70588ms to process 1 records I0504 20:39:44.092366 1 tasks_processing.go:74] worker 38 stopped. I0504 20:39:44.092457 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=68083718d2756e071dd008a54f09de12601e668ea138c9b66d9905d49121626e I0504 20:39:44.092466 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 11.648978ms to process 1 records I0504 20:39:44.092653 1 tasks_processing.go:74] worker 8 stopped. I0504 20:39:44.092665 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 12.204401ms to process 0 records I0504 20:39:44.093275 1 tasks_processing.go:74] worker 24 stopped. I0504 20:39:44.093337 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=6e1737378b2c70d4702ec5a93dba2d2818ade2f9619a3763578532e7a10e4106 I0504 20:39:44.093350 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 12.707087ms to process 1 records I0504 20:39:44.093902 1 tasks_processing.go:74] worker 50 stopped. I0504 20:39:44.093915 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 13.093381ms to process 0 records I0504 20:39:44.098541 1 tasks_processing.go:74] worker 2 stopped. I0504 20:39:44.098651 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=0a62ef59f92f88fbf84f58973147746b95718c51b3c3ddf18c010da88cbfc7f0 I0504 20:39:44.098676 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=3b5c64d674f1b138a5768ddfc4ef9ea298b6a3872866b4d6e0f8263d25479d12 I0504 20:39:44.098688 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 18.14949ms to process 2 records I0504 20:39:44.099308 1 tasks_processing.go:74] worker 41 stopped. I0504 20:39:44.099331 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0504 20:39:44.099340 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 18.395419ms to process 1 records I0504 20:39:44.101185 1 tasks_processing.go:74] worker 0 stopped. I0504 20:39:44.101300 1 recorder.go:75] Recording config/network with fingerprint=3badc857caa9570f4bd65fe5e21d64f1c7a1f6f6b50f526cf867be905e9e74f3 I0504 20:39:44.101314 1 gather.go:177] gatherer "clusterconfig" function "networks" took 20.77155ms to process 1 records I0504 20:39:44.105489 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0504 20:39:44.105572 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0504 20:39:44.105620 1 operator.go:328] started I0504 20:39:44.105652 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0504 20:39:44.107602 1 tasks_processing.go:74] worker 27 stopped. I0504 20:39:44.107619 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 26.568267ms to process 0 records I0504 20:39:44.107682 1 tasks_processing.go:74] worker 62 stopped. I0504 20:39:44.107693 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 25.869258ms to process 0 records I0504 20:39:44.120715 1 tasks_processing.go:74] worker 17 stopped. I0504 20:39:44.120726 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 38.816625ms to process 0 records I0504 20:39:44.120806 1 tasks_processing.go:74] worker 52 stopped. I0504 20:39:44.120820 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 39.172453ms to process 0 records I0504 20:39:44.121032 1 tasks_processing.go:74] worker 20 stopped. I0504 20:39:44.121048 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 39.21256ms to process 0 records I0504 20:39:44.121199 1 tasks_processing.go:74] worker 47 stopped. E0504 20:39:44.121212 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0504 20:39:44.121223 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 40.253594ms to process 0 records I0504 20:39:44.121231 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 38.938158ms to process 0 records I0504 20:39:44.121238 1 tasks_processing.go:74] worker 26 stopped. I0504 20:39:44.121250 1 tasks_processing.go:74] worker 37 stopped. I0504 20:39:44.121266 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 40.215177ms to process 0 records I0504 20:39:44.121287 1 tasks_processing.go:74] worker 63 stopped. I0504 20:39:44.121299 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 40.409872ms to process 0 records I0504 20:39:44.121307 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 39.805424ms to process 0 records I0504 20:39:44.121314 1 tasks_processing.go:74] worker 60 stopped. I0504 20:39:44.121316 1 tasks_processing.go:74] worker 34 stopped. I0504 20:39:44.121322 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 40.673129ms to process 0 records I0504 20:39:44.121327 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 40.85693ms to process 0 records I0504 20:39:44.121331 1 tasks_processing.go:74] worker 9 stopped. I0504 20:39:44.121375 1 tasks_processing.go:74] worker 48 stopped. E0504 20:39:44.121388 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0504 20:39:44.121397 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 40.563997ms to process 0 records I0504 20:39:44.121527 1 recorder.go:75] Recording config/image with fingerprint=a2c3c8707303a34d84769282b655e4f298b7db47dcc45d11cfdcb9e04414925c I0504 20:39:44.121543 1 gather.go:177] gatherer "clusterconfig" function "image" took 40.552277ms to process 1 records I0504 20:39:44.121553 1 tasks_processing.go:74] worker 23 stopped. I0504 20:39:44.121626 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=3e30bc57c8bf3a1ae6b6747fde49d805c09f564d0e141cd1ba9d6880ddb70ed6 I0504 20:39:44.121886 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=9c28c3f63732c041514d7735308a156886bb9293f10ba2b4c899e73e2e7e9e64 I0504 20:39:44.121942 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=e16810b01ab1964bdd6f526fe4c480f3a90d1c8f4b3b211b4895935ae75c89eb I0504 20:39:44.121958 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 39.636408ms to process 3 records E0504 20:39:44.121981 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0504 20:39:44.121962 1 tasks_processing.go:74] worker 51 stopped. I0504 20:39:44.121995 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 39.366677ms to process 0 records I0504 20:39:44.122036 1 tasks_processing.go:74] worker 36 stopped. I0504 20:39:44.122043 1 tasks_processing.go:74] worker 59 stopped. I0504 20:39:44.123744 1 controller.go:254] Source scaController *sca.Controller is not ready I0504 20:39:44.123758 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0504 20:39:44.123762 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0504 20:39:44.123765 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0504 20:39:44.123771 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0504 20:39:44.123819 1 controller.go:531] The operator is still being initialized I0504 20:39:44.124331 1 controller.go:554] The operator is healthy I0504 20:39:44.124421 1 recorder.go:75] Recording config/infrastructure with fingerprint=f3fc466be80ba8b2b0f307cc807f8338601963dfd78c0d7e78269afbc1091f41 I0504 20:39:44.124442 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 41.015551ms to process 1 records I0504 20:39:44.124531 1 tasks_processing.go:74] worker 39 stopped. I0504 20:39:44.124662 1 recorder.go:75] Recording config/authentication with fingerprint=7b0e0a9344d47af2880050adc3fe47fd6c4c7bfeb75d15a5eb37135cebe67a83 I0504 20:39:44.124679 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 40.913773ms to process 1 records I0504 20:39:44.124692 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 40.021418ms to process 0 records I0504 20:39:44.124698 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 40.477007ms to process 0 records I0504 20:39:44.124703 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 40.501116ms to process 0 records I0504 20:39:44.124704 1 tasks_processing.go:74] worker 35 stopped. I0504 20:39:44.124722 1 tasks_processing.go:74] worker 31 stopped. I0504 20:39:44.124726 1 tasks_processing.go:74] worker 11 stopped. I0504 20:39:44.124802 1 tasks_processing.go:74] worker 58 stopped. I0504 20:39:44.124889 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=1cbb66b960bd0da1dfc2d09303e54de5fc6d1e49c33f5630c128d75b0f267d22 I0504 20:39:44.124923 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=d808e3b504b4e4df4285e4b4896b5625fadb3dc7cddab5537d1683b2ebe8fab5 I0504 20:39:44.124949 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=574c732273070d1ffef2ac10a5d439dbab5ca0f4898343d6f570eea5e119644b I0504 20:39:44.124956 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 40.437116ms to process 3 records I0504 20:39:44.124962 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 40.852543ms to process 0 records I0504 20:39:44.124967 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 41.951742ms to process 0 records I0504 20:39:44.125011 1 tasks_processing.go:74] worker 40 stopped. I0504 20:39:44.125023 1 tasks_processing.go:74] worker 25 stopped. I0504 20:39:44.125089 1 tasks_processing.go:74] worker 13 stopped. I0504 20:39:44.125292 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=3b20f512ac3c205228303c2c2b5a35813c607aa2ecd9860d27011412f0812ef0 I0504 20:39:44.125303 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 44.135371ms to process 1 records I0504 20:39:44.125396 1 tasks_processing.go:74] worker 49 stopped. I0504 20:39:44.125448 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=dc82da636c44de1410866a561036f01c6dd9a31ea8d271031f8392b800d2cd2d I0504 20:39:44.125529 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=9009657ce81ec97836452946d4d76afa06a03e801ef9d04ef2c0ebdb59cbccaa I0504 20:39:44.125537 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 44.003508ms to process 2 records I0504 20:39:44.127859 1 prometheus_rules.go:88] Prometheus rules successfully created I0504 20:39:44.128713 1 tasks_processing.go:74] worker 7 stopped. I0504 20:39:44.128770 1 recorder.go:75] Recording config/proxy with fingerprint=8f4d5a762fe13ab0cb7d4dbf13e00a75e4179a19247a2db39bd698382c55c9a6 I0504 20:39:44.128782 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 47.13016ms to process 1 records E0504 20:39:44.131055 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27a3144f2c-a465-4047-b648-d84385256ec9%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:51121->172.30.0.10:53: read: connection refused I0504 20:39:44.131067 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27a3144f2c-a465-4047-b648-d84385256ec9%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:51121->172.30.0.10:53: read: connection refused I0504 20:39:44.131160 1 tasks_processing.go:74] worker 22 stopped. I0504 20:39:44.131366 1 recorder.go:75] Recording config/oauth with fingerprint=804eb5740a72b4eaf593a468a27c0bbcb512b6fde0fed92451653df97d45ee55 I0504 20:39:44.131378 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 49.827338ms to process 1 records I0504 20:39:44.131585 1 tasks_processing.go:74] worker 54 stopped. I0504 20:39:44.131596 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 49.044844ms to process 0 records I0504 20:39:44.131668 1 tasks_processing.go:74] worker 21 stopped. E0504 20:39:44.131680 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0504 20:39:44.131725 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0504 20:39:44.131740 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 49.821047ms to process 1 records I0504 20:39:44.131764 1 tasks_processing.go:74] worker 55 stopped. I0504 20:39:44.131876 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=04fe6e57fc9df6454bb88c04147648ddee668d1f7623be3dac8e502f4d891396 I0504 20:39:44.131953 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=c8b3374d0654952cce4ca3c9ea5041151302848100c230961a9f1e3ec0822b14 I0504 20:39:44.131971 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=3d9c5ccc2ebca88a3fc2bc1dc3df42e1fed62fbef222f26e63eebad84fece218 I0504 20:39:44.131997 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=f6389a4561caa39a92e3b878a154918787a1dfee4f912789717381aba469416d I0504 20:39:44.132019 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=91f8d93a7c3630b59897f712a8738e40f51fbfb05852c273d82dd2d1b52a77a6 I0504 20:39:44.132047 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=afb559582766f4fcd07a72adc850d917dc967ad51e9e219887e1092524be7928 I0504 20:39:44.132069 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=e0a381241ff1218e0c213efb0e3989c1b476512879df0c0165bdf1eae1588914 I0504 20:39:44.132101 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=fdeb1d3627eb183623d282969422a3fff0fadedf1f686e7a9a0e5c398140932e I0504 20:39:44.132145 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=6655293c83872b6127f3967d473ef5b2bdd69577cd5c6b46bcf728937c0c690c I0504 20:39:44.132170 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=633025c82317889c149c4efa90db6702203f9c4ea8cfb2e5499b46e838fa313e I0504 20:39:44.132191 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=96371d34f306e8c56a40ccf8e64fc7b2f116229c767a62b6eac36c5b7c73f003 I0504 20:39:44.132199 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 51.199704ms to process 11 records I0504 20:39:44.134035 1 tasks_processing.go:74] worker 44 stopped. I0504 20:39:44.134052 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 51.522404ms to process 0 records I0504 20:39:44.134354 1 tasks_processing.go:74] worker 57 stopped. I0504 20:39:44.134369 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 53.858147ms to process 0 records W0504 20:39:44.135077 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 20:39:44.139001 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0504 20:39:44.139078 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0504 20:39:44.139115 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0504 20:39:44.142086 1 tasks_processing.go:74] worker 28 stopped. I0504 20:39:44.142095 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0504 20:39:44.142617 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=729438e7f59b5f5be62efa9deb1732a6c14e8fab8271e354c9fe44ddc508ac8b I0504 20:39:44.142875 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=9fcbaeb282c6b01dae24734d6f9f587b2dec5c2d2105313e9c762da84b9440df I0504 20:39:44.142888 1 gather.go:177] gatherer "clusterconfig" function "crds" took 60.010309ms to process 2 records I0504 20:39:44.144324 1 base_controller.go:82] Caches are synced for ConfigController I0504 20:39:44.144334 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... W0504 20:39:44.145422 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:40431->172.30.0.10:53: read: connection refused I0504 20:39:44.145433 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.12:40431->172.30.0.10:53: read: connection refused I0504 20:39:44.152651 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:39:44.160202 1 tasks_processing.go:74] worker 43 stopped. E0504 20:39:44.160215 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0504 20:39:44.160221 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0504 20:39:44.160224 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0504 20:39:44.160232 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0504 20:39:44.160265 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0504 20:39:44.160271 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0504 20:39:44.160275 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0504 20:39:44.160280 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0504 20:39:44.160316 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0504 20:39:44.160323 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0504 20:39:44.160328 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 78.438953ms to process 7 records I0504 20:39:44.168098 1 tasks_processing.go:74] worker 61 stopped. E0504 20:39:44.168109 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0504 20:39:44.168115 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q3b3os70c667l7uhbfrt2sq5dfhc2b8-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q3b3os70c667l7uhbfrt2sq5dfhc2b8-primary-cert-bundle-secret" not found I0504 20:39:44.168173 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=c8f9411fda97e6f875a77f1c8151cf104bf86bab7ed052cdd3030a0c8e943cf1 I0504 20:39:44.168184 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 86.359421ms to process 1 records I0504 20:39:44.194955 1 tasks_processing.go:74] worker 42 stopped. I0504 20:39:44.194979 1 gather_logs.go:145] no pods in namespace were found I0504 20:39:44.195193 1 recorder.go:75] Recording config/version with fingerprint=682b8d64349325fa768bbb62f0b950dd7d2bfe56d19d8b1eb6de478033fe9908 I0504 20:39:44.195205 1 recorder.go:75] Recording config/id with fingerprint=aa587084b6ddb64893536e3f93906a20ec5e66f274c2a6c475474beec8dd98db I0504 20:39:44.195212 1 gather.go:177] gatherer "clusterconfig" function "version" took 113.996479ms to process 2 records I0504 20:39:44.195222 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 113.480758ms to process 0 records I0504 20:39:44.195227 1 tasks_processing.go:74] worker 14 stopped. I0504 20:39:44.205729 1 base_controller.go:82] Caches are synced for LoggingSyncer I0504 20:39:44.205750 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0504 20:39:44.208028 1 tasks_processing.go:74] worker 33 stopped. I0504 20:39:44.208088 1 recorder.go:75] Recording config/running_containers with fingerprint=1c71109572e8b07b7cea1bd036fbb378dff21c6322f288301ffbbb067f144dc1 I0504 20:39:44.208101 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 127.197316ms to process 1 records I0504 20:39:44.223907 1 tasks_processing.go:74] worker 18 stopped. I0504 20:39:44.223922 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 142.652008ms to process 0 records I0504 20:39:44.224161 1 tasks_processing.go:74] worker 10 stopped. I0504 20:39:44.224467 1 recorder.go:75] Recording config/node/ip-10-0-0-208.ec2.internal with fingerprint=de9a2cc8d2a4e4769b107fce164c1d96023706213c6ae08baedd7407d4f5588f I0504 20:39:44.224531 1 recorder.go:75] Recording config/node/ip-10-0-1-101.ec2.internal with fingerprint=4fbab40d7c9ec49f34fd650fcc518cb65287804b0105cdf1453d5321bd7707ca I0504 20:39:44.224589 1 recorder.go:75] Recording config/node/ip-10-0-2-58.ec2.internal with fingerprint=4c85791d0375922847359c5f567bdf9d4d9a75ab86cd48bf3c3eaab732ce53db I0504 20:39:44.224599 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 141.57892ms to process 3 records I0504 20:39:44.532535 1 gather_cluster_operator_pods_and_events.go:121] Found 14 pods with 15 containers I0504 20:39:44.532549 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1677721 bytes I0504 20:39:44.532609 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-9dxc7 pod in namespace openshift-dns (previous: false). I0504 20:39:44.762479 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-9dxc7 pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-9dxc7\" is waiting to start: ContainerCreating" I0504 20:39:44.762504 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-9dxc7\" is waiting to start: ContainerCreating" I0504 20:39:44.762514 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-9dxc7 pod in namespace openshift-dns (previous: false). I0504 20:39:44.939500 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-9dxc7 pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-9dxc7\" is waiting to start: ContainerCreating" I0504 20:39:44.939519 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-9dxc7\" is waiting to start: ContainerCreating" I0504 20:39:44.939529 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-6lxl9 pod in namespace openshift-dns (previous: false). W0504 20:39:45.135449 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 20:39:45.196096 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:39:45.196118 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-fcmdm pod in namespace openshift-dns (previous: false). I0504 20:39:45.340600 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:39:45.340620 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-jstqj pod in namespace openshift-dns (previous: false). I0504 20:39:45.545360 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0504 20:39:45.656752 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:39:45.656772 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-678b7c6486-8zhwb pod in namespace openshift-image-registry (previous: false). I0504 20:39:45.736660 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:39:45.736681 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7cd54b949-2zp52 pod in namespace openshift-image-registry (previous: false). I0504 20:39:45.932673 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:39:45.932695 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7cd54b949-f8shc pod in namespace openshift-image-registry (previous: false). I0504 20:39:45.946226 1 tasks_processing.go:74] worker 56 stopped. I0504 20:39:45.946279 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=43a1cf0a7eefd6feae4f3eae82e167514eeef908f80e6b5ad55f4d7d34b85d71 I0504 20:39:45.946321 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=c0ea475bdca6cc2b2ebe49b7e76b8775917af87ad68e6b941788fcb4aeb0870e I0504 20:39:45.946352 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0504 20:39:45.946376 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=fae135ff5e69d3eadc2a9f3a50c4e879068359bdedb28dda84f577775634e6ed I0504 20:39:45.946393 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0504 20:39:45.946413 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=05d849184e978f552d08bbba581338343f12a2473f8a7b87de5653254b3d0bbb I0504 20:39:45.946463 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=088bb1def78e0251af8b7e00dbcb61b31cdb1a7712fcc0d86238c135cc453671 I0504 20:39:45.946487 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=7c238efa9b3412103d7be215f7f1f11f56d7891ad5f8b01761ea861b480cc94b I0504 20:39:45.946501 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=d6e05fada9b31fd92cb4f26f206e5bc543005a1b62f8b01db57184508c877fd7 I0504 20:39:45.946517 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=a58b3ec0f01e059c505aba93cdd8b37f004e03ce9af6714b6e3001f745110bc4 I0504 20:39:45.946533 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0504 20:39:45.946550 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=b9d613450c24d8cd8956a0704d8f39a14fe18cca81209fba0c3bcac883c6f34e I0504 20:39:45.946558 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0504 20:39:45.946573 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=07d33f4888457b75fc9a8bc6c35d4df8eb5cb3ff302c981bc07e309207128246 I0504 20:39:45.946580 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0504 20:39:45.946594 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=d4c7786dc10169c5b549510b0578ef16c84f6b27f89a0183168ec55928e7ffb7 I0504 20:39:45.946603 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0504 20:39:45.946620 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=02ddc427c6844a36c30b85056f0a4a68ff01468995de993b50f0340f5f03d557 I0504 20:39:45.946736 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=ca7a2acbadc05c6f53c6cf257d14becfc870681c85677c688afecbffbe742ec9 I0504 20:39:45.946745 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0504 20:39:45.946752 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0504 20:39:45.946774 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0504 20:39:45.946796 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=2e5210aa5a01b0d5767b9e973fb1a3ff5f6f5a7eb544f52bb9d0052455e8df58 I0504 20:39:45.946820 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=ee8143ec681462c3fc9a6a8ca6e00c6b1b3dc8aaa68cc7d71303b6651718fc76 I0504 20:39:45.946829 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0504 20:39:45.946846 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=d51712f3e63635898e53d6aa311dc59e4347f9dc0a9d3bd4d0b5840bddb3c525 I0504 20:39:45.946855 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0504 20:39:45.946867 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=b6486fdb2568e920833d6187dc50c835f3dab030a14613f7f40f1e5466b55ff6 I0504 20:39:45.946883 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=0c61380654eea56fb9ff536461bd2b906f8cb8e437157da55b76810c07ea653f I0504 20:39:45.946896 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=295f016207d3be94852a37129a27ec12f94383de998d5736eae382ea64d3b66f I0504 20:39:45.946913 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=b448e3b8782b980743f0a40769807ba6e31df8c503a7e7bf203c7a8bfad58a52 I0504 20:39:45.946924 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=8c4e7ec5d729de16ed66a434496b0d6658d8e20260c7d3c633237bc34bb6dae8 I0504 20:39:45.946951 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=6a9186a6e5286c4a62f1279df1d2003f08259ed6b2efc22f72e93fdc07522cbf I0504 20:39:45.946967 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0504 20:39:45.946973 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0504 20:39:45.946980 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.86556839s to process 35 records W0504 20:39:46.136332 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 20:39:46.140414 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-7cd54b949-f8shc pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-7cd54b949-f8shc\" is waiting to start: ContainerCreating" I0504 20:39:46.140439 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-7cd54b949-f8shc\" is waiting to start: ContainerCreating" I0504 20:39:46.140451 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-2hvtd pod in namespace openshift-image-registry (previous: false). I0504 20:39:46.350369 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:39:46.350387 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-9x7dc pod in namespace openshift-image-registry (previous: false). I0504 20:39:46.539667 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:39:46.539686 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-rckfq pod in namespace openshift-image-registry (previous: false). I0504 20:39:46.741986 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0504 20:39:46.742005 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6df458665-btbqb pod in namespace openshift-ingress (previous: false). I0504 20:39:46.939509 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-6df458665-btbqb pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-6df458665-btbqb\" is waiting to start: ContainerCreating" I0504 20:39:46.939526 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-6df458665-btbqb\" is waiting to start: ContainerCreating" I0504 20:39:46.939540 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6df458665-jdgq7 pod in namespace openshift-ingress (previous: false). W0504 20:39:47.134972 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0504 20:39:47.140875 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-6df458665-jdgq7 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-6df458665-jdgq7\" is waiting to start: ContainerCreating" I0504 20:39:47.140890 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-6df458665-jdgq7\" is waiting to start: ContainerCreating" I0504 20:39:47.140901 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-858c4f66ff-jn7ws pod in namespace openshift-ingress (previous: false). I0504 20:39:47.339118 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-858c4f66ff-jn7ws pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-858c4f66ff-jn7ws\" is waiting to start: ContainerCreating" I0504 20:39:47.339160 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-858c4f66ff-jn7ws\" is waiting to start: ContainerCreating" I0504 20:39:47.339172 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-vwqn7 pod in namespace openshift-ingress-canary (previous: false). I0504 20:39:47.539789 1 tasks_processing.go:74] worker 4 stopped. I0504 20:39:47.539882 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=b4a9954484287f9c4b6dc953a761511cc305f847d9dfb798e5996901544fc549 I0504 20:39:47.539928 1 recorder.go:75] Recording events/openshift-dns with fingerprint=c5dcc74cb0ff8390caeb902a0e3d7d0e5d99837f9ae42631b3c96d3f65735672 I0504 20:39:47.540027 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=cc7d44e97655c1f39595c4e346f8057c9b1f1b6dc2ae337bb2cbdc288fbf61de I0504 20:39:47.540058 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=20a39ded7099c6dbe33b7f817948752bd64621df8f1eceb6e49450c9129a8f80 I0504 20:39:47.540110 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=8b2bca978bce1f3d3f676cedfe903fa6b425e0774f7fb093de28cfffec038be8 I0504 20:39:47.540151 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=1bc822270d9e9d00d713d0eb8f7293d1a1f5a6efda6ac9084b802683f4d69c15 I0504 20:39:47.540159 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-vwqn7/serve-healthcheck-canary_current.log with fingerprint=17982f279fcb63872f9bd8e3e1f2839d6f4a7d08223deccdeddb817db6d72e6d I0504 20:39:47.540168 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 3.458941294s to process 7 records W0504 20:39:48.136195 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0504 20:39:49.135664 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0504 20:39:49.135689 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0504 20:39:49.135705 1 tasks_processing.go:74] worker 19 stopped. E0504 20:39:49.135716 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0504 20:39:49.135727 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0504 20:39:49.135741 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0504 20:39:49.135751 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.053999818s to process 1 records I0504 20:39:56.600217 1 tasks_processing.go:74] worker 30 stopped. I0504 20:39:56.600254 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0504 20:39:56.600267 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.518952615s to process 1 records I0504 20:39:57.023180 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:39:57.288455 1 tasks_processing.go:74] worker 3 stopped. I0504 20:39:57.288742 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=698b3ed6ac2dcbbe305b75e3c37787498192c269ee2964c1d49d551d6371b6d6 I0504 20:39:57.288757 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.208061634s to process 1 records E0504 20:39:57.288812 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.209s with: function \"machines\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0504 20:39:57.289918 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machines" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "machine_healthchecks" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0504 20:39:57.289932 1 periodic.go:216] Running workloads gatherer I0504 20:39:57.289948 1 tasks_processing.go:45] number of workers: 2 I0504 20:39:57.289960 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 20:39:57.289964 1 tasks_processing.go:71] worker 1 working on workload_info task. I0504 20:39:57.289973 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 20:39:57.289993 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0504 20:39:57.316737 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 21s for image data I0504 20:39:57.320049 1 tasks_processing.go:74] worker 0 stopped. I0504 20:39:57.320067 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 30.040396ms to process 0 records I0504 20:39:57.329837 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (14ms) I0504 20:39:57.340797 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (11ms) I0504 20:39:57.352996 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (12ms) I0504 20:39:57.366964 1 gather_workloads_info.go:387] No image sha256:4733236617781e3469ffffb15e4daaa1f14ea8e1c52b426a3787a4f1f2945424 (14ms) I0504 20:39:57.383980 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (17ms) I0504 20:39:57.396912 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (13ms) I0504 20:39:57.407651 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (11ms) I0504 20:39:57.419139 1 gather_workloads_info.go:387] No image sha256:e3732e356ae2324565c74cae57d5d016917314fa293ddaa3a68ee9ae030c6f07 (11ms) I0504 20:39:57.433802 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (15ms) I0504 20:39:57.444781 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (11ms) I0504 20:39:57.456190 1 gather_workloads_info.go:387] No image sha256:4b748615a43be416d52c33df03e2cdec89ededded1d6a2ade167c5a955d56e5f (11ms) I0504 20:39:57.534983 1 gather_workloads_info.go:387] No image sha256:90a8ffd9643ebb16a6a8c04bb38cf9ed58903e9d3bf836c68f399193db5edaf6 (79ms) I0504 20:39:57.628002 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (93ms) I0504 20:39:57.728326 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (100ms) I0504 20:39:57.827835 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (99ms) I0504 20:39:57.928792 1 gather_workloads_info.go:387] No image sha256:e8b96d9318b3b8c9ed0afe4e6381f635c6b0c2f20772044ac68001ed49af2c87 (101ms) I0504 20:39:58.028704 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (100ms) I0504 20:39:58.127975 1 gather_workloads_info.go:387] No image sha256:ce138e8cf5b96557b1864ef6f27c2608bdca59be2611804366cef7169c36291e (99ms) I0504 20:39:58.227720 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (100ms) I0504 20:39:58.331559 1 gather_workloads_info.go:387] No image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (104ms) I0504 20:39:58.428490 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (97ms) I0504 20:39:58.528277 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (100ms) I0504 20:39:58.628653 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (100ms) I0504 20:39:58.727801 1 gather_workloads_info.go:387] No image sha256:d6fbe0075cbb12bfd287c973704fadf97154c7f73e370733d976a40835e9155a (99ms) I0504 20:39:58.829013 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (101ms) I0504 20:39:58.928959 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (100ms) I0504 20:39:59.027906 1 gather_workloads_info.go:387] No image sha256:e1ba458cf6f0b3606c90880da72db8ab99cd11040bae84baebe3ff2e0d1ea075 (99ms) I0504 20:39:59.128026 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (100ms) I0504 20:39:59.227868 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (100ms) I0504 20:39:59.329053 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (101ms) I0504 20:39:59.427809 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (99ms) I0504 20:39:59.528795 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (101ms) I0504 20:39:59.628096 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (99ms) I0504 20:39:59.727827 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (100ms) I0504 20:39:59.727862 1 tasks_processing.go:74] worker 1 stopped. E0504 20:39:59.727872 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0504 20:39:59.728119 1 recorder.go:75] Recording config/workload_info with fingerprint=4c8530c4317ca2e7fdeaf6d0d76d1ec4e714f1a9c6a56275be090ba73a93b2e0 I0504 20:39:59.728148 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.437890245s to process 1 records E0504 20:39:59.728178 1 periodic.go:254] "Unhandled Error" err="workloads failed after 2.438s with: function \"workload_info\" failed with an error" I0504 20:39:59.729280 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0504 20:39:59.729293 1 periodic.go:216] Running conditional gatherer I0504 20:39:59.735749 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0504 20:39:59.742793 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.12:57034->172.30.0.10:53: read: connection refused E0504 20:39:59.743032 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0504 20:39:59.743087 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0504 20:39:59.749868 1 conditional_gatherer.go:392] cluster version is '4.21.9' E0504 20:39:59.749882 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:39:59.749887 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:39:59.749890 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:39:59.749894 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:39:59.749896 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:39:59.749901 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:39:59.749904 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:39:59.749907 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0504 20:39:59.749910 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0504 20:39:59.749924 1 tasks_processing.go:45] number of workers: 3 I0504 20:39:59.749940 1 tasks_processing.go:69] worker 2 listening for tasks. I0504 20:39:59.749946 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0504 20:39:59.749948 1 tasks_processing.go:69] worker 0 listening for tasks. I0504 20:39:59.749958 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0504 20:39:59.749965 1 tasks_processing.go:69] worker 1 listening for tasks. I0504 20:39:59.749974 1 tasks_processing.go:74] worker 1 stopped. I0504 20:39:59.749983 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0504 20:39:59.750028 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0504 20:39:59.750044 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.024µs to process 1 records I0504 20:39:59.750152 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0504 20:39:59.750159 1 tasks_processing.go:74] worker 2 stopped. I0504 20:39:59.750165 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.228µs to process 1 records I0504 20:39:59.750286 1 tasks_processing.go:74] worker 0 stopped. I0504 20:39:59.750296 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 322.59µs to process 0 records I0504 20:39:59.750315 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.12:57034->172.30.0.10:53: read: connection refused I0504 20:39:59.750336 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0504 20:39:59.776076 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=fbf7b6d26314bd75e7aaa61b07c78d90fd7cc8d63757640720325be8a91a39f7 I0504 20:39:59.776225 1 diskrecorder.go:70] Writing 100 records to /var/lib/insights-operator/insights-2026-05-04-203959.tar.gz I0504 20:39:59.782002 1 diskrecorder.go:51] Wrote 100 records to disk in 5ms I0504 20:39:59.782030 1 periodic.go:285] Gathering cluster info every 2h0m0s I0504 20:39:59.782044 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0504 20:40:01.965804 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:40:02.165905 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:40:08.756606 1 configmapobserver.go:84] configmaps "insights-config" not found I0504 20:40:53.670791 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="bb7326a67085b6a9c5cfa4d1dbede4a53c452ad78969043840c9e46985b622cb") W0504 20:40:53.670825 1 builder.go:160] Restart triggered because of file /var/run/configmaps/service-ca-bundle/service-ca.crt was created I0504 20:40:53.670882 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="d5d51547310fe3253597b56b311adc5dd54e4e598cf0469d37d9d681ab39ca9f") I0504 20:40:53.670916 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0504 20:40:53.670921 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="95ac9faabd550cb83206b922a4daa61189af1ab7ce24d28070c3e056ac6281a6") I0504 20:40:53.670934 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0504 20:40:53.670957 1 base_controller.go:181] Shutting down LoggingSyncer ... I0504 20:40:53.670977 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0504 20:40:53.670983 1 base_controller.go:113] All LoggingSyncer workers have been terminated