W0514 23:14:31.891006 1 cmd.go:257] Using insecure, self-signed certificates I0514 23:14:32.443150 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0514 23:14:32.443504 1 observer_polling.go:159] Starting file observer I0514 23:14:32.698563 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0514 23:14:32.698750 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0514 23:14:32.699298 1 secure_serving.go:57] Forcing use of http/1.1 only W0514 23:14:32.699330 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0514 23:14:32.699341 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0514 23:14:32.699345 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0514 23:14:32.699347 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0514 23:14:32.699350 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0514 23:14:32.699352 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0514 23:14:32.699372 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0514 23:14:32.703145 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0514 23:14:32.703212 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"0d221d07-db5c-49bc-91f3-a3effa63ef4d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0514 23:14:32.704220 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0514 23:14:32.704238 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0514 23:14:32.704245 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0514 23:14:32.704245 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0514 23:14:32.704269 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0514 23:14:32.704272 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0514 23:14:32.704578 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-1547314408/tls.crt::/tmp/serving-cert-1547314408/tls.key" I0514 23:14:32.704867 1 secure_serving.go:213] Serving securely on [::]:8443 I0514 23:14:32.704901 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0514 23:14:32.707963 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0514 23:14:32.707990 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0514 23:14:32.708109 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0514 23:14:32.713694 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0514 23:14:32.714127 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0514 23:14:32.718576 1 secretconfigobserver.go:119] support secret does not exist I0514 23:14:32.722942 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0514 23:14:32.727622 1 secretconfigobserver.go:119] support secret does not exist I0514 23:14:32.729896 1 recorder.go:176] Pruning old reports every 7h27m41s, max age is 288h0m0s I0514 23:14:32.736442 1 periodic.go:216] Running clusterconfig gatherer I0514 23:14:32.736447 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0514 23:14:32.736462 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0514 23:14:32.736470 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0514 23:14:32.736474 1 insightsreport.go:296] Starting report retriever I0514 23:14:32.736482 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0514 23:14:32.736486 1 tasks_processing.go:45] number of workers: 64 I0514 23:14:32.736506 1 tasks_processing.go:69] worker 1 listening for tasks. I0514 23:14:32.736519 1 tasks_processing.go:69] worker 0 listening for tasks. I0514 23:14:32.736523 1 tasks_processing.go:69] worker 4 listening for tasks. I0514 23:14:32.736529 1 tasks_processing.go:71] worker 0 working on mutating_webhook_configurations task. I0514 23:14:32.736531 1 tasks_processing.go:69] worker 3 listening for tasks. I0514 23:14:32.736534 1 tasks_processing.go:71] worker 4 working on storage_cluster task. I0514 23:14:32.736538 1 tasks_processing.go:69] worker 48 listening for tasks. I0514 23:14:32.736543 1 tasks_processing.go:69] worker 6 listening for tasks. I0514 23:14:32.736546 1 tasks_processing.go:69] worker 18 listening for tasks. I0514 23:14:32.736548 1 tasks_processing.go:69] worker 40 listening for tasks. I0514 23:14:32.736552 1 tasks_processing.go:69] worker 19 listening for tasks. I0514 23:14:32.736553 1 tasks_processing.go:69] worker 7 listening for tasks. I0514 23:14:32.736558 1 tasks_processing.go:69] worker 8 listening for tasks. I0514 23:14:32.736522 1 tasks_processing.go:69] worker 2 listening for tasks. I0514 23:14:32.736528 1 tasks_processing.go:69] worker 31 listening for tasks. I0514 23:14:32.736565 1 tasks_processing.go:69] worker 42 listening for tasks. I0514 23:14:32.736566 1 tasks_processing.go:69] worker 32 listening for tasks. I0514 23:14:32.736574 1 tasks_processing.go:69] worker 9 listening for tasks. I0514 23:14:32.736574 1 tasks_processing.go:69] worker 33 listening for tasks. I0514 23:14:32.736582 1 tasks_processing.go:69] worker 10 listening for tasks. I0514 23:14:32.736574 1 tasks_processing.go:71] worker 7 working on pod_network_connectivity_checks task. I0514 23:14:32.736587 1 tasks_processing.go:69] worker 11 listening for tasks. I0514 23:14:32.736582 1 tasks_processing.go:71] worker 6 working on ingress_certificates task. I0514 23:14:32.736593 1 tasks_processing.go:69] worker 21 listening for tasks. I0514 23:14:32.736596 1 tasks_processing.go:69] worker 12 listening for tasks. I0514 23:14:32.736597 1 tasks_processing.go:69] worker 37 listening for tasks. I0514 23:14:32.736599 1 tasks_processing.go:69] worker 44 listening for tasks. I0514 23:14:32.736570 1 tasks_processing.go:71] worker 31 working on active_alerts task. I0514 23:14:32.736607 1 tasks_processing.go:71] worker 44 working on container_images task. I0514 23:14:32.736614 1 tasks_processing.go:69] worker 14 listening for tasks. I0514 23:14:32.736614 1 tasks_processing.go:69] worker 22 listening for tasks. I0514 23:14:32.736584 1 tasks_processing.go:71] worker 33 working on version task. I0514 23:14:32.736632 1 tasks_processing.go:69] worker 34 listening for tasks. W0514 23:14:32.736634 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0514 23:14:32.736634 1 tasks_processing.go:69] worker 38 listening for tasks. I0514 23:14:32.736644 1 tasks_processing.go:69] worker 35 listening for tasks. I0514 23:14:32.736648 1 tasks_processing.go:69] worker 24 listening for tasks. I0514 23:14:32.736652 1 tasks_processing.go:71] worker 40 working on monitoring_persistent_volumes task. I0514 23:14:32.736569 1 tasks_processing.go:71] worker 2 working on openstack_version task. I0514 23:14:32.736649 1 tasks_processing.go:69] worker 16 listening for tasks. I0514 23:14:32.736661 1 tasks_processing.go:69] worker 26 listening for tasks. I0514 23:14:32.736662 1 tasks_processing.go:69] worker 45 listening for tasks. I0514 23:14:32.736591 1 tasks_processing.go:69] worker 43 listening for tasks. I0514 23:14:32.736670 1 tasks_processing.go:69] worker 15 listening for tasks. I0514 23:14:32.736600 1 tasks_processing.go:71] worker 21 working on machine_healthchecks task. I0514 23:14:32.736677 1 tasks_processing.go:69] worker 29 listening for tasks. I0514 23:14:32.736683 1 tasks_processing.go:69] worker 30 listening for tasks. I0514 23:14:32.736683 1 tasks_processing.go:69] worker 46 listening for tasks. I0514 23:14:32.736553 1 tasks_processing.go:69] worker 39 listening for tasks. I0514 23:14:32.736688 1 tasks_processing.go:71] worker 12 working on certificate_signing_requests task. I0514 23:14:32.736697 1 tasks_processing.go:69] worker 60 listening for tasks. I0514 23:14:32.736708 1 tasks_processing.go:69] worker 57 listening for tasks. I0514 23:14:32.736715 1 tasks_processing.go:69] worker 58 listening for tasks. I0514 23:14:32.736723 1 tasks_processing.go:69] worker 59 listening for tasks. I0514 23:14:32.736724 1 tasks_processing.go:69] worker 56 listening for tasks. I0514 23:14:32.736738 1 tasks_processing.go:69] worker 49 listening for tasks. I0514 23:14:32.736732 1 tasks_processing.go:69] worker 52 listening for tasks. I0514 23:14:32.736614 1 tasks_processing.go:69] worker 36 listening for tasks. I0514 23:14:32.736623 1 tasks_processing.go:69] worker 23 listening for tasks. I0514 23:14:32.736577 1 tasks_processing.go:71] worker 32 working on operators task. I0514 23:14:32.736759 1 tasks_processing.go:69] worker 54 listening for tasks. I0514 23:14:32.736767 1 tasks_processing.go:71] worker 19 working on operators_pods_and_events task. I0514 23:14:32.736770 1 tasks_processing.go:69] worker 55 listening for tasks. I0514 23:14:32.736779 1 tasks_processing.go:71] worker 3 working on machine_autoscalers task. I0514 23:14:32.736833 1 tasks_processing.go:69] worker 50 listening for tasks. I0514 23:14:32.736862 1 tasks_processing.go:71] worker 50 working on sap_config task. I0514 23:14:32.736575 1 tasks_processing.go:71] worker 42 working on feature_gates task. I0514 23:14:32.736591 1 tasks_processing.go:71] worker 11 working on aggregated_monitoring_cr_names task. I0514 23:14:32.736926 1 tasks_processing.go:71] worker 14 working on image_registries task. I0514 23:14:32.736967 1 tasks_processing.go:71] worker 56 working on storage_classes task. I0514 23:14:32.737017 1 tasks_processing.go:71] worker 49 working on ingress task. I0514 23:14:32.737027 1 tasks_processing.go:71] worker 48 working on clusterroles task. I0514 23:14:32.737034 1 tasks_processing.go:71] worker 58 working on image task. I0514 23:14:32.737034 1 tasks_processing.go:71] worker 39 working on networks task. I0514 23:14:32.736606 1 tasks_processing.go:71] worker 37 working on crds task. I0514 23:14:32.737103 1 tasks_processing.go:71] worker 23 working on lokistack task. I0514 23:14:32.736582 1 tasks_processing.go:71] worker 9 working on infrastructures task. I0514 23:14:32.736535 1 tasks_processing.go:69] worker 5 listening for tasks. I0514 23:14:32.737271 1 tasks_processing.go:71] worker 5 working on support_secret task. I0514 23:14:32.736652 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 35.204µs to process 0 records I0514 23:14:32.736590 1 tasks_processing.go:71] worker 10 working on nodenetworkstates task. I0514 23:14:32.737113 1 tasks_processing.go:69] worker 62 listening for tasks. I0514 23:14:32.736565 1 tasks_processing.go:71] worker 8 working on olm_operators task. I0514 23:14:32.736653 1 tasks_processing.go:69] worker 47 listening for tasks. I0514 23:14:32.736970 1 tasks_processing.go:71] worker 34 working on openstack_dataplanenodesets task. I0514 23:14:32.737573 1 tasks_processing.go:71] worker 47 working on ceph_cluster task. I0514 23:14:32.736749 1 tasks_processing.go:69] worker 53 listening for tasks. I0514 23:14:32.736668 1 tasks_processing.go:69] worker 27 listening for tasks. I0514 23:14:32.737721 1 tasks_processing.go:71] worker 53 working on dvo_metrics task. I0514 23:14:32.736946 1 tasks_processing.go:69] worker 17 listening for tasks. I0514 23:14:32.736946 1 tasks_processing.go:71] worker 18 working on sap_datahubs task. I0514 23:14:32.737741 1 tasks_processing.go:71] worker 17 working on cost_management_metrics_configs task. I0514 23:14:32.737752 1 tasks_processing.go:71] worker 27 working on openstack_controlplanes task. I0514 23:14:32.736557 1 tasks_processing.go:69] worker 41 listening for tasks. I0514 23:14:32.737776 1 tasks_processing.go:71] worker 41 working on overlapping_namespace_uids task. I0514 23:14:32.736959 1 tasks_processing.go:71] worker 30 working on authentication task. I0514 23:14:32.736958 1 tasks_processing.go:69] worker 51 listening for tasks. I0514 23:14:32.737882 1 tasks_processing.go:71] worker 51 working on openstack_dataplanedeployments task. I0514 23:14:32.736963 1 tasks_processing.go:71] worker 22 working on container_runtime_configs task. I0514 23:14:32.736976 1 tasks_processing.go:71] worker 31 working on cluster_apiserver task. I0514 23:14:32.736980 1 tasks_processing.go:71] worker 38 working on config_maps task. I0514 23:14:32.736984 1 tasks_processing.go:71] worker 35 working on node_logs task. I0514 23:14:32.736987 1 tasks_processing.go:71] worker 24 working on jaegers task. I0514 23:14:32.736991 1 tasks_processing.go:71] worker 16 working on install_plans task. I0514 23:14:32.736994 1 tasks_processing.go:71] worker 26 working on metrics task. W0514 23:14:32.738312 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0514 23:14:32.738588 1 tasks_processing.go:71] worker 26 working on image_pruners task. I0514 23:14:32.736998 1 tasks_processing.go:71] worker 45 working on proxies task. I0514 23:14:32.738602 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 216.326µs to process 0 records I0514 23:14:32.737005 1 tasks_processing.go:71] worker 15 working on schedulers task. I0514 23:14:32.737002 1 tasks_processing.go:71] worker 43 working on oauths task. I0514 23:14:32.737008 1 tasks_processing.go:71] worker 29 working on openshift_logging task. I0514 23:14:32.737018 1 tasks_processing.go:71] worker 52 working on machine_sets task. I0514 23:14:32.737014 1 tasks_processing.go:71] worker 36 working on service_accounts task. I0514 23:14:32.737024 1 tasks_processing.go:71] worker 60 working on validating_webhook_configurations task. I0514 23:14:32.736656 1 tasks_processing.go:69] worker 25 listening for tasks. I0514 23:14:32.736585 1 tasks_processing.go:69] worker 20 listening for tasks. I0514 23:14:32.737090 1 tasks_processing.go:71] worker 57 working on tsdb_status task. I0514 23:14:32.739372 1 tasks_processing.go:71] worker 25 working on openshift_machine_api_events task. I0514 23:14:32.739435 1 tasks_processing.go:71] worker 20 working on qemu_kubevirt_launcher_logs task. I0514 23:14:32.736605 1 tasks_processing.go:69] worker 13 listening for tasks. I0514 23:14:32.737098 1 tasks_processing.go:71] worker 54 working on nodes task. I0514 23:14:32.737103 1 tasks_processing.go:71] worker 55 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0514 23:14:32.737115 1 tasks_processing.go:69] worker 63 listening for tasks. W0514 23:14:32.739455 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0514 23:14:32.739505 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 138.118µs to process 0 records I0514 23:14:32.737122 1 tasks_processing.go:71] worker 59 working on silenced_alerts task. I0514 23:14:32.739589 1 tasks_processing.go:71] worker 63 working on sap_pods task. W0514 23:14:32.739617 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0514 23:14:32.739628 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 97.282µs to process 0 records I0514 23:14:32.737129 1 tasks_processing.go:71] worker 1 working on machine_configs task. I0514 23:14:32.737121 1 tasks_processing.go:69] worker 61 listening for tasks. I0514 23:14:32.737422 1 tasks_processing.go:71] worker 62 working on machines task. I0514 23:14:32.737028 1 tasks_processing.go:71] worker 46 working on machine_config_pools task. I0514 23:14:32.736672 1 tasks_processing.go:69] worker 28 listening for tasks. I0514 23:14:32.740115 1 tasks_processing.go:74] worker 28 stopped. I0514 23:14:32.740138 1 tasks_processing.go:74] worker 61 stopped. I0514 23:14:32.740205 1 tasks_processing.go:71] worker 57 working on pdbs task. I0514 23:14:32.739794 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 3.114541ms to process 0 records I0514 23:14:32.740502 1 tasks_processing.go:71] worker 2 working on nodenetworkconfigurationpolicies task. I0514 23:14:32.740508 1 tasks_processing.go:74] worker 13 stopped. I0514 23:14:32.740464 1 tasks_processing.go:71] worker 59 working on node_features task. I0514 23:14:32.740569 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 3.9321ms to process 0 records I0514 23:14:32.740578 1 tasks_processing.go:74] worker 4 stopped. E0514 23:14:32.740756 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0514 23:14:32.740880 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 3.956332ms to process 0 records I0514 23:14:32.740782 1 tasks_processing.go:74] worker 7 stopped. I0514 23:14:32.741270 1 tasks_processing.go:74] worker 21 stopped. E0514 23:14:32.741286 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0514 23:14:32.741299 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 4.585167ms to process 0 records I0514 23:14:32.741967 1 tasks_processing.go:74] worker 3 stopped. I0514 23:14:32.741983 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 5.17833ms to process 0 records I0514 23:14:32.747289 1 tasks_processing.go:74] worker 23 stopped. I0514 23:14:32.747305 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 10.122788ms to process 0 records I0514 23:14:32.753255 1 tasks_processing.go:74] worker 50 stopped. I0514 23:14:32.753270 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 16.383398ms to process 0 records I0514 23:14:32.753280 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 15.952979ms to process 0 records I0514 23:14:32.753287 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 15.703683ms to process 0 records I0514 23:14:32.753292 1 tasks_processing.go:74] worker 10 stopped. I0514 23:14:32.753296 1 tasks_processing.go:74] worker 47 stopped. I0514 23:14:32.753314 1 tasks_processing.go:74] worker 52 stopped. I0514 23:14:32.753325 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 14.271564ms to process 0 records I0514 23:14:32.753331 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 15.543321ms to process 0 records I0514 23:14:32.753335 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 12.80181ms to process 0 records I0514 23:14:32.753339 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 13.297066ms to process 0 records I0514 23:14:32.753342 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 15.580957ms to process 0 records I0514 23:14:32.753347 1 tasks_processing.go:74] worker 17 stopped. I0514 23:14:32.753351 1 tasks_processing.go:74] worker 27 stopped. I0514 23:14:32.753354 1 tasks_processing.go:74] worker 2 stopped. I0514 23:14:32.753358 1 tasks_processing.go:74] worker 46 stopped. I0514 23:14:32.753475 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0514 23:14:32.753492 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0514 23:14:32.753496 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0514 23:14:32.753500 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0514 23:14:32.753515 1 controller.go:531] The operator is still being initialized I0514 23:14:32.753523 1 controller.go:554] The operator is healthy I0514 23:14:32.763858 1 tasks_processing.go:74] worker 58 stopped. I0514 23:14:32.763979 1 recorder.go:75] Recording config/image with fingerprint=002889b9bd2c009aae31155ea5b6ef9e53edca4d2c7f8f9c58100a32012e30e6 I0514 23:14:32.763995 1 gather.go:177] gatherer "clusterconfig" function "image" took 26.813861ms to process 1 records E0514 23:14:32.764007 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0514 23:14:32.764015 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 26.642368ms to process 0 records I0514 23:14:32.764020 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 24.398264ms to process 0 records I0514 23:14:32.764025 1 tasks_processing.go:74] worker 63 stopped. I0514 23:14:32.764034 1 tasks_processing.go:74] worker 5 stopped. I0514 23:14:32.764202 1 tasks_processing.go:74] worker 24 stopped. E0514 23:14:32.764216 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0514 23:14:32.764216 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 26.065824ms to process 0 records I0514 23:14:32.764320 1 tasks_processing.go:74] worker 9 stopped. I0514 23:14:32.764719 1 recorder.go:75] Recording config/infrastructure with fingerprint=6d1bb0bff8f2b7c1e44580d2f23be99bf7d87be77533584d1ef8ef71211c9b3e I0514 23:14:32.764732 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 26.956066ms to process 1 records I0514 23:14:32.764744 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 26.287547ms to process 0 records I0514 23:14:32.764748 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 26.645581ms to process 0 records I0514 23:14:32.764751 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 23.638609ms to process 0 records E0514 23:14:32.764756 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0514 23:14:32.764761 1 gather.go:177] gatherer "clusterconfig" function "machines" took 24.449658ms to process 0 records I0514 23:14:32.764768 1 tasks_processing.go:74] worker 22 stopped. I0514 23:14:32.764778 1 tasks_processing.go:74] worker 59 stopped. I0514 23:14:32.764781 1 tasks_processing.go:74] worker 34 stopped. I0514 23:14:32.764782 1 tasks_processing.go:74] worker 62 stopped. I0514 23:14:32.764849 1 tasks_processing.go:74] worker 39 stopped. I0514 23:14:32.764879 1 recorder.go:75] Recording config/network with fingerprint=4a7b16f19eaf2cb1fe42edff497c39afabfa9273452c6827738a7c15a213a0bf I0514 23:14:32.764888 1 gather.go:177] gatherer "clusterconfig" function "networks" took 27.580421ms to process 1 records I0514 23:14:32.768790 1 tasks_processing.go:74] worker 14 stopped. I0514 23:14:32.769245 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=0c02b33938513ba80defd8f5f84f7fc41806abd0a5bf9ff0564819789e8c21a3 I0514 23:14:32.769259 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 31.813138ms to process 1 records I0514 23:14:32.781305 1 tasks_processing.go:74] worker 18 stopped. I0514 23:14:32.781324 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 43.566717ms to process 0 records I0514 23:14:32.781363 1 tasks_processing.go:74] worker 51 stopped. I0514 23:14:32.781380 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 43.459029ms to process 0 records I0514 23:14:32.781651 1 tasks_processing.go:74] worker 49 stopped. I0514 23:14:32.781827 1 recorder.go:75] Recording config/ingress with fingerprint=d76f72d8c412a9f3556983739bc2c0dfbf0c0f5362319d4ae3dad14741f326b3 I0514 23:14:32.781844 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 44.614842ms to process 1 records E0514 23:14:32.781853 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0514 23:14:32.781871 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0514 23:14:32.781880 1 tasks_processing.go:74] worker 1 stopped. I0514 23:14:32.781884 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 42.129767ms to process 1 records I0514 23:14:32.781892 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 43.66775ms to process 0 records I0514 23:14:32.781900 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 45.206739ms to process 0 records I0514 23:14:32.781907 1 tasks_processing.go:74] worker 40 stopped. I0514 23:14:32.781913 1 tasks_processing.go:74] worker 35 stopped. I0514 23:14:32.781997 1 tasks_processing.go:74] worker 29 stopped. I0514 23:14:32.782013 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 43.089072ms to process 0 records I0514 23:14:32.782098 1 tasks_processing.go:74] worker 26 stopped. I0514 23:14:32.782132 1 gather_logs.go:145] no pods in namespace were found I0514 23:14:32.782142 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=8b86266a6eda4243ef62c9c29b97a80255ac256747f6fdd08314b7154172fa1d I0514 23:14:32.782153 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 43.37941ms to process 1 records I0514 23:14:32.782266 1 tasks_processing.go:74] worker 43 stopped. I0514 23:14:32.782434 1 recorder.go:75] Recording config/oauth with fingerprint=22c1e17c6929b5b28b3f26b05f72e8bae549ed380f0ba7ac8a21aad5cf5c04b1 I0514 23:14:32.782445 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 43.170651ms to process 1 records I0514 23:14:32.782494 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=79cc5cda23780562682f266f4ed79194293cce8df7f6e4cf32f75ae1c4396e00 I0514 23:14:32.782505 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 43.297785ms to process 1 records I0514 23:14:32.782510 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 42.353447ms to process 0 records I0514 23:14:32.782526 1 tasks_processing.go:74] worker 15 stopped. I0514 23:14:32.782533 1 tasks_processing.go:74] worker 20 stopped. I0514 23:14:32.782548 1 recorder.go:75] Recording config/proxy with fingerprint=c22cbe1aaabbae8b6b4d16cb5a124865afc9eaa3640af9299bd71a64d1c32bc1 I0514 23:14:32.782552 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 43.523034ms to process 1 records I0514 23:14:32.782598 1 tasks_processing.go:74] worker 45 stopped. I0514 23:14:32.782631 1 recorder.go:75] Recording config/featuregate with fingerprint=317e009d99ad69f70345c0e44cd2425c331ed841c2b3feb08a3755c25cd33326 I0514 23:14:32.782638 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 45.395813ms to process 1 records I0514 23:14:32.782653 1 tasks_processing.go:74] worker 42 stopped. I0514 23:14:32.782719 1 tasks_processing.go:74] worker 30 stopped. I0514 23:14:32.782768 1 recorder.go:75] Recording config/authentication with fingerprint=97769df141471d622aa137c6e96006ab1ed4f94a75262af63c219512ec030787 I0514 23:14:32.782776 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 44.421395ms to process 1 records I0514 23:14:32.782862 1 tasks_processing.go:74] worker 31 stopped. I0514 23:14:32.782867 1 recorder.go:75] Recording config/apiserver with fingerprint=1b4e8407a75ec0daaf6025492d148296603d992dbceca8221ea286aa03420561 I0514 23:14:32.782873 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 44.28394ms to process 1 records I0514 23:14:32.783524 1 tasks_processing.go:74] worker 25 stopped. I0514 23:14:32.783535 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 44.13814ms to process 0 records I0514 23:14:32.784484 1 tasks_processing.go:74] worker 54 stopped. I0514 23:14:32.785317 1 recorder.go:75] Recording config/node/ip-10-0-0-78.ec2.internal with fingerprint=976522ce1eebf5a04bb05925ddbc3b66e97629a0c0feb953a4a0e0783f3e8ace I0514 23:14:32.785653 1 recorder.go:75] Recording config/node/ip-10-0-1-98.ec2.internal with fingerprint=beb676654066682b69d22ed2322a67970d9f1ceb4e29b33d704346d901f93f65 I0514 23:14:32.785817 1 recorder.go:75] Recording config/node/ip-10-0-2-152.ec2.internal with fingerprint=192fe8299fee123a12927fbaaf8a21fdeceeea3575f3bea09a76a0f1b60b3f5a I0514 23:14:32.785835 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 44.752232ms to process 3 records I0514 23:14:32.785993 1 tasks_processing.go:74] worker 0 stopped. I0514 23:14:32.786163 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=ef84d044af2b73dc331be5588bcea918a6f43cafd506ab8beb1010546228d6c7 I0514 23:14:32.786294 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=db44be8b573889a0c532a7bbb01c6e6c42c31931f89fdca9aea711767941d8a3 I0514 23:14:32.786364 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=690c321e47497e3dd634543b175c896db8723792b760684a8077d1dd2b2f3349 I0514 23:14:32.786401 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 48.027633ms to process 3 records I0514 23:14:32.786552 1 tasks_processing.go:74] worker 60 stopped. I0514 23:14:32.786598 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=296014017237681e1dd445ba5deb59bc0810f67eda5b2ec03cde1c4a2dcb54e7 I0514 23:14:32.786748 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=7e35aa082b954fccd8905199199ab52badd900cbd40b4ac04303751cf5165296 I0514 23:14:32.786783 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=5dd6fc4df64ec8bca6eba617c3fbbbdb4cba0319c7f02609951016f8f345cc9f I0514 23:14:32.786836 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=17e4b625ff70d8378684152ab18e68947df2361447a7102dfd7b2baee0b298b2 I0514 23:14:32.786938 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=2cbafdaf50bdca62db8b3b679e87ca7876e9fe3106648b7c87b2ab655303313d I0514 23:14:32.787057 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=60bffa8d220709d7b4bb9b7903190dfd6e1e19f3b0155b5079a83055af7ad523 I0514 23:14:32.787102 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=d5f75669d97137e61c2a8aa7ec36cf507b0c32c2272dba0a2db7f2344b9bc9c3 I0514 23:14:32.787221 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=f5db7fc1f87b8e3c18ebcd2216083721f492d07f181d73abcfc046780a237ce3 I0514 23:14:32.787666 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=9e02867df30e5c6be20e6840155eacfd1f09f88dcfb01d2b9896d1223e10aa27 I0514 23:14:32.787721 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=8a0e15a4796193baa59a9b6a2dd5053a468f9e447782c325ac96f865980d5e8b I0514 23:14:32.787802 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=62fe66eac1024c3afb63c82575019f233833634841abea30f100dc46c38f65ec I0514 23:14:32.787811 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 47.027588ms to process 11 records I0514 23:14:32.789037 1 tasks_processing.go:74] worker 12 stopped. I0514 23:14:32.789059 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 52.338058ms to process 0 records I0514 23:14:32.790819 1 tasks_processing.go:74] worker 41 stopped. E0514 23:14:32.790832 1 gather.go:140] gatherer "clusterconfig" function "overlapping_namespace_uids" failed with the error: can't read uid range of the openshift-kube-storage-version-migrator namespace I0514 23:14:32.790850 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0514 23:14:32.790858 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 53.028526ms to process 1 records I0514 23:14:32.796845 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0514 23:14:32.796865 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0514 23:14:32.797060 1 operator.go:328] started I0514 23:14:32.797085 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0514 23:14:32.798568 1 tasks_processing.go:74] worker 44 stopped. I0514 23:14:32.798631 1 recorder.go:75] Recording config/running_containers with fingerprint=0e6ecf9e86a157ca863f3acf0df6277c6c76875b6c331cda4e2476ca23109dd8 I0514 23:14:32.798644 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 61.947402ms to process 1 records I0514 23:14:32.805201 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0514 23:14:32.805277 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0514 23:14:32.805349 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0514 23:14:32.808332 1 base_controller.go:82] Caches are synced for ConfigController I0514 23:14:32.808344 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0514 23:14:32.812139 1 tasks_processing.go:74] worker 37 stopped. I0514 23:14:32.812669 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=40c508537e8b36f0da4e90a9cae6114fa892e859695f6bf39e1489245646abab I0514 23:14:32.812886 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=ac7c9637dc3d8ed35f513becac69f64125d845fe8f7c4718f4b01e01c7e90ad9 I0514 23:14:32.812899 1 gather.go:177] gatherer "clusterconfig" function "crds" took 75.020739ms to process 2 records I0514 23:14:32.827657 1 tasks_processing.go:74] worker 11 stopped. I0514 23:14:32.827675 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 90.756542ms to process 0 records I0514 23:14:32.827780 1 tasks_processing.go:74] worker 57 stopped. I0514 23:14:32.827919 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=345c37c4ef0d5f574065bbd9a082dcda6fad2e0aac1410ba1abef894b2bea682 I0514 23:14:32.827950 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=bed4da59bbe264e770146f34dffc65221de009af729e986bc99e32a1ddc009b3 I0514 23:14:32.827976 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=753ee834a7adbd2e52653e2e0d7614f07b3d6a7d255c230616a88b4c30c5f10f I0514 23:14:32.827986 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 87.536693ms to process 3 records I0514 23:14:32.828069 1 tasks_processing.go:74] worker 48 stopped. I0514 23:14:32.828163 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=768b8d39bd5ff33801e0d65ba064f84702c1d49137f296f5b842cb048773d8e4 I0514 23:14:32.828281 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=4bd0149b97e655f7be5ffa4f1068c8c1e448bd2eef6478df49372a5415c85ed1 I0514 23:14:32.828291 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 90.785539ms to process 2 records I0514 23:14:32.828356 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=ea7403f561fc8cbb7fe3fdc66626f00e426f6e6bc67c779f4629018590c1d40c I0514 23:14:32.828370 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=11f51e83eec72cc63635ebef676306a6c09125608172036949fdc9a1cf12ba7f I0514 23:14:32.828375 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 90.864775ms to process 2 records I0514 23:14:32.828378 1 tasks_processing.go:74] worker 56 stopped. I0514 23:14:32.828379 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 90.482035ms to process 0 records I0514 23:14:32.828384 1 tasks_processing.go:74] worker 8 stopped. I0514 23:14:32.829603 1 prometheus_rules.go:88] Prometheus rules successfully created W0514 23:14:32.832292 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0514 23:14:32.834989 1 configmapobserver.go:84] configmaps "insights-config" not found E0514 23:14:32.839117 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27348a8854-0a8d-435a-8146-b88ebf471bd0%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:37047->172.30.0.10:53: read: connection refused I0514 23:14:32.839133 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27348a8854-0a8d-435a-8146-b88ebf471bd0%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:37047->172.30.0.10:53: read: connection refused I0514 23:14:32.843683 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0514 23:14:32.843695 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0514 23:14:32.843701 1 controller.go:254] Source scaController *sca.Controller is not ready I0514 23:14:32.843705 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0514 23:14:32.843728 1 controller.go:531] The operator is still being initialized I0514 23:14:32.843736 1 controller.go:554] The operator is healthy I0514 23:14:32.879355 1 tasks_processing.go:74] worker 33 stopped. I0514 23:14:32.879652 1 recorder.go:75] Recording config/version with fingerprint=3b2db316d345ff504119552c361707dfcfe73efbf74611cfbb192de77c7e45d9 I0514 23:14:32.879667 1 recorder.go:75] Recording config/id with fingerprint=36ab9eeefd115cbc6ba7f9700ffe47ac28861d6a7a325e1c199cb2068b274a2a I0514 23:14:32.879674 1 gather.go:177] gatherer "clusterconfig" function "version" took 142.720016ms to process 2 records I0514 23:14:32.895568 1 tasks_processing.go:74] worker 38 stopped. E0514 23:14:32.895589 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0514 23:14:32.895596 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0514 23:14:32.895601 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0514 23:14:32.895615 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0514 23:14:32.895667 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0514 23:14:32.895679 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0514 23:14:32.895686 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0514 23:14:32.895693 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0514 23:14:32.895758 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0514 23:14:32.895769 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0514 23:14:32.895778 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 157.484954ms to process 7 records I0514 23:14:32.897966 1 base_controller.go:82] Caches are synced for LoggingSyncer I0514 23:14:32.897978 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0514 23:14:32.903747 1 tasks_processing.go:74] worker 55 stopped. I0514 23:14:32.903767 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 164.013361ms to process 0 records I0514 23:14:32.908858 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0514 23:14:32.909125 1 tasks_processing.go:74] worker 6 stopped. E0514 23:14:32.909139 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0514 23:14:32.909150 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2qa0bb9570tfkc4jihthla6gq803g80l-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2qa0bb9570tfkc4jihthla6gq803g80l-primary-cert-bundle-secret" not found I0514 23:14:32.909245 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=8c7040750b26960f218090eb8640f8af555dbf8c6fc075f2c5de4ccd79c85ef3 I0514 23:14:32.909264 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 172.523501ms to process 1 records W0514 23:14:32.912736 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:34555->172.30.0.10:53: read: connection refused I0514 23:14:32.912750 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:34555->172.30.0.10:53: read: connection refused I0514 23:14:32.948262 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0514 23:14:32.964916 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0514 23:14:33.197042 1 gather_cluster_operator_pods_and_events.go:121] Found 14 pods with 15 containers I0514 23:14:33.197057 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1677721 bytes I0514 23:14:33.197799 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-gcr6r pod in namespace openshift-dns (previous: false). I0514 23:14:33.402259 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-gcr6r pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-gcr6r\" is waiting to start: ContainerCreating" I0514 23:14:33.402275 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-gcr6r\" is waiting to start: ContainerCreating" I0514 23:14:33.402286 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-gcr6r pod in namespace openshift-dns (previous: false). I0514 23:14:33.599885 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-gcr6r pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-gcr6r\" is waiting to start: ContainerCreating" I0514 23:14:33.599905 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-gcr6r\" is waiting to start: ContainerCreating" I0514 23:14:33.599917 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-47rwv pod in namespace openshift-dns (previous: false). I0514 23:14:33.800018 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0514 23:14:33.800036 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-694kt pod in namespace openshift-dns (previous: false). W0514 23:14:33.831757 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0514 23:14:34.035615 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0514 23:14:34.035633 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-l2p66 pod in namespace openshift-dns (previous: false). I0514 23:14:34.224021 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0514 23:14:34.224047 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-55bc57d65c-h99kj pod in namespace openshift-image-registry (previous: false). I0514 23:14:34.235491 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0514 23:14:34.400131 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-55bc57d65c-h99kj pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-55bc57d65c-h99kj\" is waiting to start: ContainerCreating" I0514 23:14:34.400151 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-55bc57d65c-h99kj\" is waiting to start: ContainerCreating" I0514 23:14:34.400165 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-55bc57d65c-mbggh pod in namespace openshift-image-registry (previous: false). I0514 23:14:34.595445 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0514 23:14:34.595462 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-854969bbd6-gtc27 pod in namespace openshift-image-registry (previous: false). I0514 23:14:34.638866 1 tasks_processing.go:74] worker 32 stopped. I0514 23:14:34.638924 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=5a3464f4acd20b6eab3761ebda036d67ae2127774f7f7b942cf9f88f21157b3a I0514 23:14:34.638970 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=41fd817a131e69ae4bb5c585f272116cd14f02b6e0341855688cea257b87c920 I0514 23:14:34.639002 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0514 23:14:34.639027 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=4843ab02067899d8fd8b8a6ab70846eaad11d57d67ee4d305fcf77ab8d80775c I0514 23:14:34.639045 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0514 23:14:34.639066 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=a5c14506d291ab651b18c418af09ad2a7a1bf1249e42aad9ee710054576c6f9e I0514 23:14:34.639116 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=3a5fc22240450dd6cd42e976c43b7cc49f3022ecd372dd1af8545fed31061e8f I0514 23:14:34.639142 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=c6206ceafb6081a23832a32a92a246375331f939f960f2b88c70a442164701a0 I0514 23:14:34.639155 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=486c70be1fb8b75298ab66dbe95ea5e49a4c190104c5a6cf42630e038732b3e2 I0514 23:14:34.639192 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=1ca00b12b0da6d8ecab1dd97e09225f58340758a058fb541c06b3f98af1bd9be I0514 23:14:34.639203 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0514 23:14:34.639220 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=a6676437f823c00725844dc9eea32f1a0ac458579619dbe51b162b60d49582d0 I0514 23:14:34.639229 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0514 23:14:34.639245 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=53bef0cc9c12c54886fa25fb609ee93ea4ef1b2177514268f37e0d5d7460e131 I0514 23:14:34.639252 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0514 23:14:34.639270 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=c08815a4a4051be83ed80de3f87739f87645b218d4dbeb882320c9790f1a8081 I0514 23:14:34.639281 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0514 23:14:34.639296 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=bfe6f47edb33b389a32392ea3d32088befb92189b4336d461d1c6d8e05e5f608 I0514 23:14:34.639420 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=91f10e2aaf87c0d2366fc2033daf7265f6966b77816a086dc48ff43e4d67d83b I0514 23:14:34.639430 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0514 23:14:34.639438 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0514 23:14:34.639460 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0514 23:14:34.639483 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=94e1247c5daceae8cdbc217b87e0c36bb288c85e50fd3371e792f6e30382f81e I0514 23:14:34.639506 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=09b367895d5814759c5b91680b52eaf9874d723c05867ed7d5e318fbe29081b5 I0514 23:14:34.639515 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0514 23:14:34.639531 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=9af336dc94e0ce8c6aa605619b774870ff21749ee464dd7a9081ff425d3615b4 I0514 23:14:34.639540 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0514 23:14:34.639554 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=46948139f590b10629c256e4181458a3041d6155f7fa64fa796c3777c8e80aec I0514 23:14:34.639568 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=60994d7ef25a95bf73f4869d87e8d3110cecc041d682946867def4303b56cb04 I0514 23:14:34.639581 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=ef7051c6ae5bbabeae39f8e1e3241adb41852be90d1aefb30cda4f1ff7c898f7 I0514 23:14:34.639597 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=ab42003b8bf62ffdbb46f619258fcad846e1572e911adbc713356a50b1cbce8f I0514 23:14:34.639607 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=582b41dca7536c3650a26b39c954d705846e89bfa305192787070c1b8dcc23be I0514 23:14:34.639635 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=08429f43d1e5587d08f263812566a6c2201d9edf9ea58f736fcd23197240ee21 I0514 23:14:34.639651 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0514 23:14:34.639658 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0514 23:14:34.639665 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.902100095s to process 35 records I0514 23:14:34.795915 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0514 23:14:34.795934 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-9q6jf pod in namespace openshift-image-registry (previous: false). W0514 23:14:34.831840 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0514 23:14:35.004058 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0514 23:14:35.004076 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-tbncl pod in namespace openshift-image-registry (previous: false). I0514 23:14:35.205672 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0514 23:14:35.205691 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-vgrkf pod in namespace openshift-image-registry (previous: false). I0514 23:14:35.401832 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0514 23:14:35.401849 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-778445446b-fvv77 pod in namespace openshift-ingress (previous: false). I0514 23:14:35.600911 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-778445446b-fvv77 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-778445446b-fvv77\" is waiting to start: ContainerCreating" I0514 23:14:35.600930 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-778445446b-fvv77\" is waiting to start: ContainerCreating" I0514 23:14:35.600942 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-778445446b-nzlw9 pod in namespace openshift-ingress (previous: false). I0514 23:14:35.806748 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-778445446b-nzlw9 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-778445446b-nzlw9\" is waiting to start: ContainerCreating" I0514 23:14:35.806766 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-778445446b-nzlw9\" is waiting to start: ContainerCreating" I0514 23:14:35.806778 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7f4c9f6df5-xjsx9 pod in namespace openshift-ingress (previous: false). W0514 23:14:35.831450 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0514 23:14:36.000978 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-7f4c9f6df5-xjsx9 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-7f4c9f6df5-xjsx9\" is waiting to start: ContainerCreating" I0514 23:14:36.000996 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-7f4c9f6df5-xjsx9\" is waiting to start: ContainerCreating" I0514 23:14:36.001007 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-9zk7z pod in namespace openshift-ingress-canary (previous: false). I0514 23:14:36.204105 1 tasks_processing.go:74] worker 19 stopped. I0514 23:14:36.204228 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=cbb19ee2a9f76ee7e0d5a21d7325986fb488ead4019b9a9e18e5cbf84015252b I0514 23:14:36.204279 1 recorder.go:75] Recording events/openshift-dns with fingerprint=0b331a8b7088f1336b04cc0a844893f7239fe616419ab76ca2f7837176bf5d0a I0514 23:14:36.204362 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=65dcb4aff2c4d54762535c9f3c75d4277a5fa036bee18a23dd849853574dd5b7 I0514 23:14:36.204389 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=1a5d951f03f9b3c28f80357b6faf32e45a0fdbadca702a6f2b9525c1f9a236ca I0514 23:14:36.204434 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=f3e6b8891cc72679c338076ab1bdc2f9df258855df867a11b134933934286ac2 I0514 23:14:36.204454 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=86e9af79e2d7dc616f9225d5aa343a8b6617f6130edfeedd473dbb0c7bee187e I0514 23:14:36.204462 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-9zk7z/serve-healthcheck-canary_current.log with fingerprint=420895d7f72fc91104cf53ba0d5c606e4c4b01973e4ac9d2d5a97725c85fc2ac I0514 23:14:36.204469 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 3.467321428s to process 7 records W0514 23:14:36.831581 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0514 23:14:37.832072 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0514 23:14:37.832111 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0514 23:14:37.832127 1 tasks_processing.go:74] worker 53 stopped. E0514 23:14:37.832138 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0514 23:14:37.832149 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0514 23:14:37.832163 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0514 23:14:37.832185 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.094389363s to process 1 records I0514 23:14:45.597986 1 tasks_processing.go:74] worker 16 stopped. I0514 23:14:45.598023 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0514 23:14:45.598035 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.859769557s to process 1 records I0514 23:14:45.758974 1 configmapobserver.go:84] configmaps "insights-config" not found I0514 23:14:46.348288 1 tasks_processing.go:74] worker 36 stopped. I0514 23:14:46.348572 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=055c7b7c45504ca3e51cba546a8ab24ab747770543cb057134d38bfd847fc95f I0514 23:14:46.348588 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.60912633s to process 1 records E0514 23:14:46.348653 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.612s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"machines\" failed with an error, function \"machine_configs\" failed with an error, function \"overlapping_namespace_uids\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0514 23:14:46.349761 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "machines" failed with an error, function "machine_configs" failed with an error, function "overlapping_namespace_uids" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0514 23:14:46.349775 1 periodic.go:216] Running workloads gatherer I0514 23:14:46.349790 1 tasks_processing.go:45] number of workers: 2 I0514 23:14:46.349799 1 tasks_processing.go:69] worker 1 listening for tasks. I0514 23:14:46.349804 1 tasks_processing.go:71] worker 1 working on workload_info task. I0514 23:14:46.349870 1 tasks_processing.go:69] worker 0 listening for tasks. I0514 23:14:46.349948 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0514 23:14:46.374587 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 22s for image data I0514 23:14:46.383530 1 gather_workloads_info.go:387] No image sha256:3a1c7ec9bb0db343348170a70697ea3209ed81e22f1259ff694190e5d04ef250 (10ms) I0514 23:14:46.391659 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (8ms) I0514 23:14:46.399765 1 gather_workloads_info.go:387] No image sha256:c886c2a85f0c920fcd6927a7c76644a422cf77f07b185120a8021e0e98d6acb9 (8ms) I0514 23:14:46.407918 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (8ms) I0514 23:14:46.415791 1 gather_workloads_info.go:387] No image sha256:4733236617781e3469ffffb15e4daaa1f14ea8e1c52b426a3787a4f1f2945424 (8ms) I0514 23:14:46.424423 1 gather_workloads_info.go:387] No image sha256:d6fbe0075cbb12bfd287c973704fadf97154c7f73e370733d976a40835e9155a (9ms) I0514 23:14:46.432077 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (8ms) I0514 23:14:46.439691 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (8ms) I0514 23:14:46.447585 1 gather_workloads_info.go:387] No image sha256:e3732e356ae2324565c74cae57d5d016917314fa293ddaa3a68ee9ae030c6f07 (8ms) I0514 23:14:46.456169 1 gather_workloads_info.go:387] No image sha256:e8b96d9318b3b8c9ed0afe4e6381f635c6b0c2f20772044ac68001ed49af2c87 (9ms) I0514 23:14:46.457157 1 tasks_processing.go:74] worker 0 stopped. I0514 23:14:46.457190 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 107.192353ms to process 0 records I0514 23:14:46.482486 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (26ms) I0514 23:14:46.571912 1 configmapobserver.go:84] configmaps "insights-config" not found I0514 23:14:46.582643 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (100ms) I0514 23:14:46.684360 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (102ms) I0514 23:14:46.770016 1 configmapobserver.go:84] configmaps "insights-config" not found I0514 23:14:46.784313 1 gather_workloads_info.go:387] No image sha256:4b748615a43be416d52c33df03e2cdec89ededded1d6a2ade167c5a955d56e5f (100ms) I0514 23:14:46.883204 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (99ms) I0514 23:14:46.983242 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (100ms) I0514 23:14:47.083860 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (101ms) I0514 23:14:47.183029 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (99ms) I0514 23:14:47.283740 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (101ms) I0514 23:14:47.383103 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (99ms) I0514 23:14:47.484025 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (101ms) I0514 23:14:47.583791 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (100ms) I0514 23:14:47.683070 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (99ms) I0514 23:14:47.784003 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (101ms) I0514 23:14:47.883192 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (99ms) I0514 23:14:47.983529 1 gather_workloads_info.go:387] No image sha256:e1ba458cf6f0b3606c90880da72db8ab99cd11040bae84baebe3ff2e0d1ea075 (100ms) I0514 23:14:48.084202 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (101ms) I0514 23:14:48.183628 1 gather_workloads_info.go:387] No image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (99ms) I0514 23:14:48.283383 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (100ms) I0514 23:14:48.383714 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (100ms) I0514 23:14:48.484339 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (101ms) I0514 23:14:48.583629 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (99ms) I0514 23:14:48.684160 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (101ms) I0514 23:14:48.784024 1 gather_workloads_info.go:387] No image sha256:ce138e8cf5b96557b1864ef6f27c2608bdca59be2611804366cef7169c36291e (100ms) I0514 23:14:48.883929 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (100ms) I0514 23:14:48.883960 1 tasks_processing.go:74] worker 1 stopped. E0514 23:14:48.883985 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0514 23:14:48.884277 1 recorder.go:75] Recording config/workload_info with fingerprint=e9aee6bece97655e3c82d6a18634625ce9c974263b6816f810bc0f266107c61c I0514 23:14:48.884298 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.534148784s to process 1 records E0514 23:14:48.884334 1 periodic.go:254] "Unhandled Error" err="workloads failed after 2.534s with: function \"workload_info\" failed with an error" I0514 23:14:48.885440 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0514 23:14:48.885451 1 periodic.go:216] Running conditional gatherer I0514 23:14:48.891449 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0514 23:14:48.898259 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.13:52093->172.30.0.10:53: read: connection refused E0514 23:14:48.898517 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0514 23:14:48.898587 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0514 23:14:48.904545 1 conditional_gatherer.go:392] cluster version is '4.21.9' E0514 23:14:48.904561 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0514 23:14:48.904568 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0514 23:14:48.904573 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0514 23:14:48.904577 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0514 23:14:48.904582 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0514 23:14:48.904586 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0514 23:14:48.904590 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0514 23:14:48.904595 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0514 23:14:48.904599 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0514 23:14:48.904624 1 tasks_processing.go:45] number of workers: 3 I0514 23:14:48.904640 1 tasks_processing.go:69] worker 2 listening for tasks. I0514 23:14:48.904647 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0514 23:14:48.904650 1 tasks_processing.go:69] worker 0 listening for tasks. I0514 23:14:48.904659 1 tasks_processing.go:69] worker 1 listening for tasks. I0514 23:14:48.904668 1 tasks_processing.go:74] worker 1 stopped. I0514 23:14:48.904661 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0514 23:14:48.904672 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0514 23:14:48.904740 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0514 23:14:48.904757 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.558µs to process 1 records I0514 23:14:48.904804 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0514 23:14:48.904817 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.142µs to process 1 records I0514 23:14:48.904826 1 tasks_processing.go:74] worker 0 stopped. I0514 23:14:48.904932 1 tasks_processing.go:74] worker 2 stopped. I0514 23:14:48.904943 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 247.865µs to process 0 records I0514 23:14:48.904971 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.13:52093->172.30.0.10:53: read: connection refused I0514 23:14:48.904989 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0514 23:14:48.927738 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=e9ae8ea99d396189ee10e005bd9e3c33a87b91721a5bf58a05beac8c931decc3 I0514 23:14:48.927877 1 diskrecorder.go:70] Writing 100 records to /var/lib/insights-operator/insights-2026-05-14-231448.tar.gz I0514 23:14:48.934010 1 diskrecorder.go:51] Wrote 100 records to disk in 6ms I0514 23:14:48.934041 1 periodic.go:285] Gathering cluster info every 2h0m0s I0514 23:14:48.934055 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0514 23:14:59.117539 1 configmapobserver.go:84] configmaps "insights-config" not found I0514 23:16:02.444670 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="85067f9eac97a4fb4d3b2faf29038035d7891da20cb904c044796a1f422987c1") W0514 23:16:02.444707 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.key was created I0514 23:16:02.444763 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0514 23:16:02.444786 1 periodic.go:177] Shutting down I0514 23:16:02.444798 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="033cec6be48c53556ebc5982a194bb333611411dea81c85e1f649f089c9a68f8")