W0512 03:00:35.789709 1 cmd.go:257] Using insecure, self-signed certificates I0512 03:00:36.544990 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0512 03:00:36.545288 1 observer_polling.go:159] Starting file observer I0512 03:00:36.837668 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0512 03:00:36.837862 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0512 03:00:36.838436 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0512 03:00:36.838934 1 secure_serving.go:57] Forcing use of http/1.1 only W0512 03:00:36.838958 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0512 03:00:36.838968 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0512 03:00:36.838975 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0512 03:00:36.838980 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0512 03:00:36.838984 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0512 03:00:36.838988 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0512 03:00:36.843009 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0512 03:00:36.843043 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"3fa6bd28-02b2-4dee-a119-013b3e70f47c", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0512 03:00:36.845379 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0512 03:00:36.845400 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0512 03:00:36.845396 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0512 03:00:36.845399 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0512 03:00:36.845411 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0512 03:00:36.845419 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0512 03:00:36.845752 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-1815537684/tls.crt::/tmp/serving-cert-1815537684/tls.key" I0512 03:00:36.846255 1 secure_serving.go:213] Serving securely on [::]:8443 I0512 03:00:36.846289 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0512 03:00:36.852293 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0512 03:00:36.852321 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0512 03:00:36.852429 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0512 03:00:36.858459 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0512 03:00:36.858482 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0512 03:00:36.864077 1 secretconfigobserver.go:119] support secret does not exist I0512 03:00:36.869773 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0512 03:00:36.875367 1 secretconfigobserver.go:119] support secret does not exist I0512 03:00:36.880199 1 recorder.go:161] Pruning old reports every 4h3m5s, max age is 288h0m0s I0512 03:00:36.886524 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0512 03:00:36.886542 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0512 03:00:36.886550 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0512 03:00:36.886565 1 insightsreport.go:296] Starting report retriever I0512 03:00:36.886573 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0512 03:00:36.886589 1 periodic.go:209] Running clusterconfig gatherer I0512 03:00:36.886642 1 tasks_processing.go:45] number of workers: 64 I0512 03:00:36.886665 1 tasks_processing.go:69] worker 1 listening for tasks. I0512 03:00:36.886674 1 tasks_processing.go:69] worker 0 listening for tasks. I0512 03:00:36.886679 1 tasks_processing.go:69] worker 6 listening for tasks. I0512 03:00:36.886684 1 tasks_processing.go:69] worker 4 listening for tasks. I0512 03:00:36.886687 1 tasks_processing.go:69] worker 3 listening for tasks. I0512 03:00:36.886692 1 tasks_processing.go:69] worker 23 listening for tasks. I0512 03:00:36.886693 1 tasks_processing.go:69] worker 14 listening for tasks. I0512 03:00:36.886700 1 tasks_processing.go:69] worker 2 listening for tasks. I0512 03:00:36.886699 1 tasks_processing.go:69] worker 7 listening for tasks. I0512 03:00:36.886706 1 tasks_processing.go:69] worker 19 listening for tasks. I0512 03:00:36.886707 1 tasks_processing.go:69] worker 8 listening for tasks. I0512 03:00:36.886705 1 tasks_processing.go:69] worker 5 listening for tasks. I0512 03:00:36.886713 1 tasks_processing.go:69] worker 12 listening for tasks. I0512 03:00:36.886723 1 tasks_processing.go:71] worker 12 working on openshift_logging task. I0512 03:00:36.886723 1 tasks_processing.go:69] worker 9 listening for tasks. I0512 03:00:36.886723 1 tasks_processing.go:69] worker 10 listening for tasks. I0512 03:00:36.886722 1 tasks_processing.go:71] worker 5 working on ceph_cluster task. I0512 03:00:36.886725 1 tasks_processing.go:69] worker 18 listening for tasks. I0512 03:00:36.886716 1 tasks_processing.go:71] worker 19 working on nodes task. I0512 03:00:36.886739 1 tasks_processing.go:69] worker 21 listening for tasks. I0512 03:00:36.886741 1 tasks_processing.go:69] worker 15 listening for tasks. I0512 03:00:36.886736 1 tasks_processing.go:69] worker 13 listening for tasks. I0512 03:00:36.886747 1 tasks_processing.go:69] worker 22 listening for tasks. I0512 03:00:36.886715 1 tasks_processing.go:69] worker 11 listening for tasks. I0512 03:00:36.886757 1 tasks_processing.go:69] worker 27 listening for tasks. I0512 03:00:36.886757 1 tasks_processing.go:69] worker 38 listening for tasks. I0512 03:00:36.886760 1 tasks_processing.go:71] worker 9 working on openstack_dataplanenodesets task. I0512 03:00:36.886764 1 tasks_processing.go:69] worker 50 listening for tasks. I0512 03:00:36.886772 1 tasks_processing.go:71] worker 50 working on version task. I0512 03:00:36.886773 1 tasks_processing.go:69] worker 44 listening for tasks. I0512 03:00:36.886774 1 tasks_processing.go:69] worker 47 listening for tasks. I0512 03:00:36.886774 1 tasks_processing.go:69] worker 33 listening for tasks. I0512 03:00:36.886782 1 tasks_processing.go:69] worker 34 listening for tasks. I0512 03:00:36.886738 1 tasks_processing.go:69] worker 26 listening for tasks. I0512 03:00:36.886716 1 tasks_processing.go:71] worker 8 working on networks task. I0512 03:00:36.886789 1 tasks_processing.go:69] worker 45 listening for tasks. I0512 03:00:36.886792 1 tasks_processing.go:69] worker 35 listening for tasks. I0512 03:00:36.886795 1 tasks_processing.go:69] worker 39 listening for tasks. I0512 03:00:36.886798 1 tasks_processing.go:69] worker 46 listening for tasks. I0512 03:00:36.886751 1 tasks_processing.go:69] worker 37 listening for tasks. I0512 03:00:36.886804 1 tasks_processing.go:71] worker 10 working on clusterroles task. I0512 03:00:36.886757 1 tasks_processing.go:71] worker 22 working on sap_pods task. I0512 03:00:36.886802 1 tasks_processing.go:69] worker 32 listening for tasks. I0512 03:00:36.886816 1 tasks_processing.go:69] worker 62 listening for tasks. I0512 03:00:36.886817 1 tasks_processing.go:69] worker 42 listening for tasks. I0512 03:00:36.886824 1 tasks_processing.go:69] worker 29 listening for tasks. I0512 03:00:36.886827 1 tasks_processing.go:69] worker 51 listening for tasks. I0512 03:00:36.886811 1 tasks_processing.go:69] worker 41 listening for tasks. I0512 03:00:36.886834 1 tasks_processing.go:69] worker 30 listening for tasks. I0512 03:00:36.886836 1 tasks_processing.go:69] worker 57 listening for tasks. I0512 03:00:36.886831 1 tasks_processing.go:69] worker 56 listening for tasks. I0512 03:00:36.886844 1 tasks_processing.go:69] worker 31 listening for tasks. I0512 03:00:36.886846 1 tasks_processing.go:69] worker 53 listening for tasks. I0512 03:00:36.886847 1 tasks_processing.go:71] worker 15 working on container_runtime_configs task. I0512 03:00:36.886851 1 tasks_processing.go:69] worker 59 listening for tasks. I0512 03:00:36.886855 1 tasks_processing.go:69] worker 55 listening for tasks. I0512 03:00:36.886856 1 tasks_processing.go:71] worker 13 working on ingress task. I0512 03:00:36.886855 1 tasks_processing.go:69] worker 54 listening for tasks. I0512 03:00:36.886862 1 tasks_processing.go:69] worker 60 listening for tasks. I0512 03:00:36.886731 1 tasks_processing.go:69] worker 20 listening for tasks. I0512 03:00:36.886766 1 tasks_processing.go:69] worker 28 listening for tasks. I0512 03:00:36.886734 1 tasks_processing.go:69] worker 16 listening for tasks. I0512 03:00:36.886765 1 tasks_processing.go:69] worker 17 listening for tasks. I0512 03:00:36.886784 1 tasks_processing.go:69] worker 48 listening for tasks. I0512 03:00:36.886878 1 tasks_processing.go:71] worker 1 working on nodenetworkstates task. I0512 03:00:36.886879 1 tasks_processing.go:69] worker 61 listening for tasks. I0512 03:00:36.886787 1 tasks_processing.go:69] worker 40 listening for tasks. I0512 03:00:36.886800 1 tasks_processing.go:71] worker 21 working on metrics task. I0512 03:00:36.886913 1 tasks_processing.go:71] worker 0 working on olm_operators task. I0512 03:00:36.886844 1 tasks_processing.go:69] worker 58 listening for tasks. I0512 03:00:36.886806 1 tasks_processing.go:71] worker 18 working on machine_healthchecks task. W0512 03:00:36.886935 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0512 03:00:36.886751 1 tasks_processing.go:69] worker 49 listening for tasks. I0512 03:00:36.886827 1 tasks_processing.go:69] worker 36 listening for tasks. I0512 03:00:36.886765 1 tasks_processing.go:71] worker 38 working on machine_configs task. I0512 03:00:36.886732 1 tasks_processing.go:69] worker 25 listening for tasks. I0512 03:00:36.886874 1 tasks_processing.go:69] worker 63 listening for tasks. I0512 03:00:36.886888 1 tasks_processing.go:71] worker 6 working on qemu_kubevirt_launcher_logs task. I0512 03:00:36.887031 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 49.558µs to process 0 records I0512 03:00:36.886778 1 tasks_processing.go:69] worker 43 listening for tasks. I0512 03:00:36.886839 1 tasks_processing.go:69] worker 52 listening for tasks. I0512 03:00:36.887102 1 tasks_processing.go:71] worker 2 working on config_maps task. I0512 03:00:36.886765 1 tasks_processing.go:71] worker 27 working on jaegers task. I0512 03:00:36.887148 1 tasks_processing.go:71] worker 7 working on lokistack task. I0512 03:00:36.886868 1 tasks_processing.go:71] worker 3 working on schedulers task. I0512 03:00:36.886724 1 tasks_processing.go:69] worker 24 listening for tasks. I0512 03:00:36.886759 1 tasks_processing.go:71] worker 11 working on certificate_signing_requests task. I0512 03:00:36.886977 1 tasks_processing.go:71] worker 4 working on machine_autoscalers task. I0512 03:00:36.887430 1 tasks_processing.go:71] worker 31 working on silenced_alerts task. I0512 03:00:36.887439 1 tasks_processing.go:71] worker 53 working on sap_config task. I0512 03:00:36.887466 1 tasks_processing.go:71] worker 49 working on operators task. W0512 03:00:36.887469 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0512 03:00:36.887474 1 tasks_processing.go:71] worker 60 working on oauths task. I0512 03:00:36.887482 1 tasks_processing.go:74] worker 31 stopped. I0512 03:00:36.887493 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 40.132µs to process 0 records I0512 03:00:36.887502 1 tasks_processing.go:71] worker 63 working on pdbs task. I0512 03:00:36.887507 1 tasks_processing.go:71] worker 47 working on openstack_controlplanes task. I0512 03:00:36.887573 1 tasks_processing.go:71] worker 36 working on install_plans task. I0512 03:00:36.887613 1 tasks_processing.go:71] worker 23 working on overlapping_namespace_uids task. I0512 03:00:36.887621 1 tasks_processing.go:71] worker 14 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0512 03:00:36.887666 1 tasks_processing.go:71] worker 43 working on service_accounts task. I0512 03:00:36.887685 1 tasks_processing.go:71] worker 20 working on pod_network_connectivity_checks task. I0512 03:00:36.887750 1 tasks_processing.go:71] worker 61 working on sap_datahubs task. I0512 03:00:36.887753 1 tasks_processing.go:71] worker 55 working on validating_webhook_configurations task. I0512 03:00:36.887854 1 tasks_processing.go:71] worker 25 working on storage_cluster task. I0512 03:00:36.888502 1 tasks_processing.go:71] worker 33 working on operators_pods_and_events task. I0512 03:00:36.888506 1 tasks_processing.go:71] worker 52 working on aggregated_monitoring_cr_names task. I0512 03:00:36.888541 1 tasks_processing.go:71] worker 40 working on monitoring_persistent_volumes task. I0512 03:00:36.887503 1 tasks_processing.go:71] worker 44 working on crds task. I0512 03:00:36.888855 1 tasks_processing.go:71] worker 24 working on infrastructures task. I0512 03:00:36.888984 1 tasks_processing.go:71] worker 41 working on container_images task. I0512 03:00:36.889002 1 tasks_processing.go:71] worker 21 working on tsdb_status task. I0512 03:00:36.889009 1 tasks_processing.go:71] worker 58 working on mutating_webhook_configurations task. I0512 03:00:36.889018 1 tasks_processing.go:71] worker 62 working on proxies task. I0512 03:00:36.889019 1 tasks_processing.go:71] worker 16 working on nodenetworkconfigurationpolicies task. I0512 03:00:36.889026 1 tasks_processing.go:71] worker 59 working on authentication task. W0512 03:00:36.889027 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0512 03:00:36.889039 1 tasks_processing.go:74] worker 21 stopped. I0512 03:00:36.889053 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 26.297µs to process 0 records I0512 03:00:36.889124 1 tasks_processing.go:71] worker 34 working on support_secret task. I0512 03:00:36.889132 1 tasks_processing.go:71] worker 17 working on ingress_certificates task. I0512 03:00:36.889137 1 tasks_processing.go:71] worker 42 working on storage_classes task. I0512 03:00:36.889191 1 tasks_processing.go:71] worker 29 working on image_registries task. I0512 03:00:36.889243 1 tasks_processing.go:71] worker 57 working on openshift_machine_api_events task. I0512 03:00:36.889248 1 tasks_processing.go:71] worker 30 working on cost_management_metrics_configs task. I0512 03:00:36.889255 1 tasks_processing.go:71] worker 45 working on cluster_apiserver task. I0512 03:00:36.889266 1 tasks_processing.go:71] worker 51 working on machine_config_pools task. I0512 03:00:36.889293 1 tasks_processing.go:71] worker 28 working on openstack_dataplanedeployments task. I0512 03:00:36.887433 1 tasks_processing.go:71] worker 48 working on image task. I0512 03:00:36.889298 1 tasks_processing.go:71] worker 46 working on openstack_version task. I0512 03:00:36.889300 1 tasks_processing.go:71] worker 37 working on dvo_metrics task. I0512 03:00:36.889309 1 tasks_processing.go:71] worker 26 working on image_pruners task. I0512 03:00:36.889334 1 tasks_processing.go:71] worker 39 working on machines task. I0512 03:00:36.889353 1 tasks_processing.go:71] worker 56 working on feature_gates task. I0512 03:00:36.889375 1 tasks_processing.go:71] worker 32 working on active_alerts task. I0512 03:00:36.889380 1 tasks_processing.go:71] worker 35 working on node_logs task. W0512 03:00:36.890309 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0512 03:00:36.890641 1 tasks_processing.go:74] worker 32 stopped. I0512 03:00:36.889386 1 tasks_processing.go:71] worker 54 working on machine_sets task. I0512 03:00:36.890640 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 65.253µs to process 0 records I0512 03:00:36.892435 1 tasks_processing.go:74] worker 12 stopped. I0512 03:00:36.892450 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 5.698352ms to process 0 records I0512 03:00:36.892464 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 5.660522ms to process 0 records I0512 03:00:36.892472 1 tasks_processing.go:74] worker 9 stopped. I0512 03:00:36.892560 1 tasks_processing.go:74] worker 18 stopped. E0512 03:00:36.892569 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0512 03:00:36.892576 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 5.62005ms to process 0 records I0512 03:00:36.893472 1 tasks_processing.go:74] worker 27 stopped. I0512 03:00:36.893485 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 6.313693ms to process 0 records I0512 03:00:36.895238 1 tasks_processing.go:74] worker 7 stopped. I0512 03:00:36.895254 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 8.071426ms to process 0 records I0512 03:00:36.895260 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 8.380867ms to process 0 records I0512 03:00:36.895268 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 7.823546ms to process 0 records I0512 03:00:36.895275 1 tasks_processing.go:74] worker 4 stopped. I0512 03:00:36.895280 1 tasks_processing.go:74] worker 15 stopped. I0512 03:00:36.895504 1 tasks_processing.go:74] worker 53 stopped. I0512 03:00:36.895517 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 8.055463ms to process 0 records I0512 03:00:36.896254 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0512 03:00:36.896270 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0512 03:00:36.896275 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0512 03:00:36.896278 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0512 03:00:36.896293 1 controller.go:489] The operator is still being initialized I0512 03:00:36.896300 1 controller.go:512] The operator is healthy I0512 03:00:36.897508 1 tasks_processing.go:74] worker 13 stopped. I0512 03:00:36.897695 1 recorder.go:75] Recording config/ingress with fingerprint=b70214ffb2141edcf45afdbe6662065dad9c5795d1ae6fda1bb5a789504600fe I0512 03:00:36.897709 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 10.639575ms to process 1 records I0512 03:00:36.898759 1 tasks_processing.go:74] worker 22 stopped. I0512 03:00:36.898771 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 11.94079ms to process 0 records I0512 03:00:36.900532 1 tasks_processing.go:74] worker 5 stopped. I0512 03:00:36.900542 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 13.791531ms to process 0 records I0512 03:00:36.912114 1 tasks_processing.go:74] worker 61 stopped. I0512 03:00:36.912131 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 24.351185ms to process 0 records I0512 03:00:36.912139 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 25.243485ms to process 0 records I0512 03:00:36.912146 1 tasks_processing.go:74] worker 1 stopped. I0512 03:00:36.912421 1 tasks_processing.go:74] worker 8 stopped. I0512 03:00:36.912615 1 recorder.go:75] Recording config/network with fingerprint=0dc157da3d7ea18db6c3a39402b00f11d708c3cdd592966cb0617a48924f137b I0512 03:00:36.912633 1 gather.go:177] gatherer "clusterconfig" function "networks" took 25.624382ms to process 1 records I0512 03:00:36.912674 1 tasks_processing.go:74] worker 47 stopped. I0512 03:00:36.912687 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 25.15903ms to process 0 records I0512 03:00:36.913045 1 tasks_processing.go:74] worker 60 stopped. I0512 03:00:36.913098 1 gather_logs.go:145] no pods in namespace were found I0512 03:00:36.913409 1 recorder.go:75] Recording config/oauth with fingerprint=26b3febf81e73040bf48badd9c76277f0034fd0782820cb0693641894a10c6d5 I0512 03:00:36.913429 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 25.559103ms to process 1 records E0512 03:00:36.913439 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0512 03:00:36.913448 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 25.421843ms to process 0 records I0512 03:00:36.913456 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 26.147435ms to process 0 records I0512 03:00:36.913462 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 24.885081ms to process 0 records I0512 03:00:36.913467 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 24.801589ms to process 0 records I0512 03:00:36.913472 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 23.732242ms to process 0 records I0512 03:00:36.913480 1 tasks_processing.go:74] worker 30 stopped. I0512 03:00:36.913486 1 tasks_processing.go:74] worker 20 stopped. I0512 03:00:36.913491 1 tasks_processing.go:74] worker 6 stopped. I0512 03:00:36.913496 1 tasks_processing.go:74] worker 25 stopped. I0512 03:00:36.913500 1 tasks_processing.go:74] worker 40 stopped. I0512 03:00:36.913509 1 tasks_processing.go:74] worker 3 stopped. I0512 03:00:36.913593 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=e94a73521974a6c9c118239ce59ab88cf44ba256016a60e270d61cab3930b7b9 I0512 03:00:36.913613 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 26.224925ms to process 1 records I0512 03:00:36.913621 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 23.684353ms to process 0 records I0512 03:00:36.913629 1 tasks_processing.go:74] worker 28 stopped. I0512 03:00:36.913635 1 tasks_processing.go:74] worker 62 stopped. I0512 03:00:36.913712 1 recorder.go:75] Recording config/proxy with fingerprint=73bcdaa3948e211d27fb43d934a6a1f7434c747cc13897135ad8fb80f007fe56 I0512 03:00:36.913730 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 24.607659ms to process 1 records I0512 03:00:36.916582 1 tasks_processing.go:74] worker 54 stopped. I0512 03:00:36.916594 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 25.918139ms to process 0 records I0512 03:00:36.916658 1 tasks_processing.go:74] worker 19 stopped. I0512 03:00:36.916926 1 recorder.go:75] Recording config/node/ip-10-0-1-251.ec2.internal with fingerprint=dcc04f0d811a06fb3ef7145259e20408f2a581e88aa3459636c4dce86c733045 I0512 03:00:36.916938 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 29.908234ms to process 1 records I0512 03:00:36.917478 1 tasks_processing.go:74] worker 46 stopped. I0512 03:00:36.917488 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 27.414267ms to process 0 records I0512 03:00:36.917497 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 28.452985ms to process 0 records I0512 03:00:36.917502 1 tasks_processing.go:74] worker 16 stopped. I0512 03:00:36.919588 1 tasks_processing.go:74] worker 63 stopped. I0512 03:00:36.919685 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=51f1d68c293a22b7d1c2ba1a8cb48233d70be850426173b9c1ffc000eae921db I0512 03:00:36.919715 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=7c143ffd1b2277b5c0c2757ef337e12e24f3ac2397140e8eb64c03e129995246 I0512 03:00:36.919742 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=9bc6fafef76aedb9cd14a1dd544ed68840609de4dfe9f546b9371532f36719d7 I0512 03:00:36.919760 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 32.073316ms to process 3 records I0512 03:00:36.925779 1 tasks_processing.go:74] worker 11 stopped. I0512 03:00:36.925792 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 38.426082ms to process 0 records I0512 03:00:36.928149 1 tasks_processing.go:74] worker 57 stopped. I0512 03:00:36.928176 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 38.552904ms to process 0 records I0512 03:00:36.928226 1 tasks_processing.go:74] worker 39 stopped. E0512 03:00:36.928246 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0512 03:00:36.928258 1 gather.go:177] gatherer "clusterconfig" function "machines" took 38.023461ms to process 0 records I0512 03:00:36.928313 1 tasks_processing.go:74] worker 34 stopped. E0512 03:00:36.928364 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0512 03:00:36.928382 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 38.908832ms to process 0 records I0512 03:00:36.928492 1 tasks_processing.go:74] worker 42 stopped. I0512 03:00:36.928542 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=4aafd6c97718c0d2921295b6ac96a827f0076c2fcd7e63276b3faf68b755897a I0512 03:00:36.928574 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=aed2f31c53e13ec01ae01cdd09f4a6b98a1c56e68b61855b594ccbf4dbddf639 I0512 03:00:36.928583 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 38.81383ms to process 2 records I0512 03:00:36.928773 1 tasks_processing.go:74] worker 59 stopped. I0512 03:00:36.929346 1 recorder.go:75] Recording config/authentication with fingerprint=00e05349803650f719a2a9e1ffcf3505e40ada30c81a7517d8327fda7961b706 I0512 03:00:36.929365 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 39.736004ms to process 1 records I0512 03:00:36.930649 1 tasks_processing.go:74] worker 48 stopped. I0512 03:00:36.930720 1 recorder.go:75] Recording config/image with fingerprint=11200121b8f93083a63275f9b42b56adb6a6641ff22e3079a9a1f1758df799a7 I0512 03:00:36.930732 1 gather.go:177] gatherer "clusterconfig" function "image" took 40.607356ms to process 1 records I0512 03:00:36.930810 1 tasks_processing.go:74] worker 24 stopped. I0512 03:00:36.931209 1 recorder.go:75] Recording config/infrastructure with fingerprint=f0979a4cbea7d341dee023004a5b8f23dc9ecb0cca2da0110dddbbbf629e9d18 I0512 03:00:36.931220 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 41.781899ms to process 1 records I0512 03:00:36.931714 1 tasks_processing.go:74] worker 26 stopped. I0512 03:00:36.931913 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=af09ea18aa876f130c7b6335863e89386a30c44602c5b754ceff1fdb5213758d I0512 03:00:36.931929 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 41.574053ms to process 1 records I0512 03:00:36.932166 1 tasks_processing.go:74] worker 58 stopped. I0512 03:00:36.932388 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=dac8e32dd12809147864952b06a546fd968b7f71406a7cfca4007d2fd65ff3db I0512 03:00:36.932453 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=556a9a8bcdbca3220fe7c444e413a02866b66798ab9c6672e25580b32ddc0be0 I0512 03:00:36.932546 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=0d49a29b97b50138f7f4db339081c68b88665312ba30425b057072646a5f963f I0512 03:00:36.932556 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 43.138719ms to process 3 records I0512 03:00:36.932586 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0512 03:00:36.932599 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 44.541413ms to process 1 records I0512 03:00:36.932659 1 tasks_processing.go:74] worker 23 stopped. I0512 03:00:36.932714 1 recorder.go:75] Recording config/featuregate with fingerprint=246269b36cae33fd7a55448dfca1438b52c10b9f22ce0a3b00c1cabdd71a83a7 I0512 03:00:36.932729 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 41.974024ms to process 1 records I0512 03:00:36.932755 1 tasks_processing.go:74] worker 56 stopped. I0512 03:00:36.932819 1 tasks_processing.go:74] worker 55 stopped. I0512 03:00:36.932869 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=b75226decd84b440e19e58cf5864ab690744bb9a0ead00534a4ec26676917681 I0512 03:00:36.933020 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=001964af52341736daf9f57fed2e69d7f6d80eeef4cf2d29ec77164f41509564 I0512 03:00:36.933054 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=bb991ee2e648ad85facbd6d23c382bc6603e8d0bad0c72e02f61b25a9fdda46b I0512 03:00:36.933105 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=a06b555e283c1739cf14437062dd089aaa0b2e7a185683d7b9c525e27834b123 I0512 03:00:36.933153 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=1de45170aa59e425257821fad4058b7dde93826c392d0dab61529b95ffbf9c92 I0512 03:00:36.933224 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=316167bb9d7ee12dd9c81e9ee51de13ce140f19f6b39e926503bfa3235f514c9 I0512 03:00:36.933282 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=6c8e23b2b8fe2804e347e27d395ce9dddbdd78456af3e154704600c51232de3e I0512 03:00:36.933357 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=edc4c86ff7563f860897486b1b47b1717c34049450957217d1fe1553d624319a I0512 03:00:36.933396 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=f47f92e9dfebf2bec7b4f8e08b16af9b75002ec57f7673730a11ad097c8bad94 I0512 03:00:36.933433 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=a9881ca6116101403200d29c728bfbc517aba6c96b46b569f919dd2b7c0b916d I0512 03:00:36.933462 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=b2727eb14d9c29d4351cac8ea45724200507b62446a90699d4f3c010e1169aac I0512 03:00:36.933470 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 44.523907ms to process 11 records I0512 03:00:36.933562 1 tasks_processing.go:74] worker 45 stopped. I0512 03:00:36.933611 1 recorder.go:75] Recording config/apiserver with fingerprint=5301470ad18f5b380bc4c331e643f44f64aa336773060f9d4bd62b49fda01849 I0512 03:00:36.933637 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 42.61774ms to process 1 records I0512 03:00:36.933647 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 43.941963ms to process 0 records I0512 03:00:36.934285 1 tasks_processing.go:74] worker 52 stopped. I0512 03:00:36.934347 1 tasks_processing.go:74] worker 10 stopped. I0512 03:00:36.934456 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=31e39b58f46a3d645b4d6b8c0d1f5243810d8dd2d4cd6f656875e9840ddd597d I0512 03:00:36.934606 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=38e36a48c2474af4bbf9f94679aa817084f625dc9498f1cc1075598353f3c146 I0512 03:00:36.934629 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 45.718076ms to process 2 records I0512 03:00:36.934925 1 tasks_processing.go:74] worker 29 stopped. I0512 03:00:36.935460 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=bcfd1b77a761b12be5de32730156f246bf0624346f6c2e361e1ce37589b2068a I0512 03:00:36.935484 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 43.300444ms to process 1 records I0512 03:00:36.935576 1 recorder.go:75] Recording config/olm_operators with fingerprint=b9514cee28614f07b3cdafbf9be7e7cddbb50f457f35901d5093968db9fc07ac I0512 03:00:36.935596 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 46.19361ms to process 1 records I0512 03:00:36.935607 1 tasks_processing.go:74] worker 0 stopped. I0512 03:00:36.937571 1 tasks_processing.go:74] worker 44 stopped. I0512 03:00:36.938058 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=9c1e58f9fcce708df2c5d7cc6781be03b4754fb9ce503a897e069dae76599b62 I0512 03:00:36.938262 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=eece8486e0aeb59b798d3f7a082df6949de4a7ef69f9c59ce0f8c2d8e700976f I0512 03:00:36.938272 1 gather.go:177] gatherer "clusterconfig" function "crds" took 48.961432ms to process 2 records I0512 03:00:36.938461 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0512 03:00:36.938498 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0512 03:00:36.938621 1 operator.go:288] started I0512 03:00:36.938650 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer W0512 03:00:36.938753 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0512 03:00:36.946062 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0512 03:00:36.946075 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0512 03:00:36.946083 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0512 03:00:36.949176 1 tasks_processing.go:74] worker 14 stopped. I0512 03:00:36.949189 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 61.531246ms to process 0 records I0512 03:00:36.951407 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0512 03:00:36.951435 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0512 03:00:36.951441 1 controller.go:212] Source scaController *sca.Controller is not ready I0512 03:00:36.951444 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0512 03:00:36.951447 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0512 03:00:36.951463 1 controller.go:489] The operator is still being initialized I0512 03:00:36.951471 1 controller.go:512] The operator is healthy I0512 03:00:36.952574 1 base_controller.go:82] Caches are synced for ConfigController I0512 03:00:36.952586 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0512 03:00:36.955788 1 prometheus_rules.go:88] Prometheus rules successfully created E0512 03:00:36.958189 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%2768873eb4-3a55-47cc-8a47-664a767e9907%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:57314->172.30.0.10:53: read: connection refused I0512 03:00:36.958206 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%2768873eb4-3a55-47cc-8a47-664a767e9907%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:57314->172.30.0.10:53: read: connection refused I0512 03:00:36.965240 1 configmapobserver.go:84] configmaps "insights-config" not found I0512 03:00:36.972161 1 tasks_processing.go:74] worker 2 stopped. E0512 03:00:36.972175 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0512 03:00:36.972180 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0512 03:00:36.972184 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0512 03:00:36.972194 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0512 03:00:36.972219 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0512 03:00:36.972226 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0512 03:00:36.972230 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0512 03:00:36.972234 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0512 03:00:36.972270 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0512 03:00:36.972278 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0512 03:00:36.972283 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 85.044906ms to process 7 records I0512 03:00:36.998319 1 tasks_processing.go:74] worker 51 stopped. I0512 03:00:36.998334 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 108.523258ms to process 0 records I0512 03:00:37.004746 1 tasks_processing.go:74] worker 35 stopped. I0512 03:00:37.004759 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 114.440987ms to process 0 records I0512 03:00:37.005066 1 tasks_processing.go:74] worker 50 stopped. I0512 03:00:37.005427 1 recorder.go:75] Recording config/version with fingerprint=e2f94ab0b212e0641aeb81b1662f0785012ebe8b8e1d1ec39cbec3965989e235 I0512 03:00:37.005444 1 recorder.go:75] Recording config/id with fingerprint=72697f43ea81c79b7aebb83faca0d2e9f3472c2bd4dd9a09f855ed72b7e969a3 I0512 03:00:37.005451 1 gather.go:177] gatherer "clusterconfig" function "version" took 118.283518ms to process 2 records I0512 03:00:37.010146 1 tasks_processing.go:74] worker 38 stopped. I0512 03:00:37.010171 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0512 03:00:37.010180 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 123.153538ms to process 1 records I0512 03:00:37.012165 1 tasks_processing.go:74] worker 41 stopped. I0512 03:00:37.012233 1 recorder.go:75] Recording config/running_containers with fingerprint=3b0124ba0bef8279824bd1da65856756a9baba2452f530182005207ca32ef663 I0512 03:00:37.012247 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 123.160699ms to process 1 records I0512 03:00:37.018770 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0512 03:00:37.021914 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:57585->172.30.0.10:53: read: connection refused I0512 03:00:37.021926 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:57585->172.30.0.10:53: read: connection refused I0512 03:00:37.039741 1 base_controller.go:82] Caches are synced for LoggingSyncer I0512 03:00:37.039754 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0512 03:00:37.043460 1 tasks_processing.go:74] worker 17 stopped. E0512 03:00:37.043473 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0512 03:00:37.043479 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q849d8j6fs4l0vid19bea5pdr4lvpfb-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q849d8j6fs4l0vid19bea5pdr4lvpfb-primary-cert-bundle-secret" not found I0512 03:00:37.043559 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=a3a42acdcd81eed66e551dfd437d9a2d3ce512f13d23712f44c91214d47a75ed I0512 03:00:37.043576 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 153.978548ms to process 1 records I0512 03:00:37.342775 1 gather_cluster_operator_pods_and_events.go:121] Found 10 pods with 11 containers I0512 03:00:37.342791 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 2287802 bytes I0512 03:00:37.343438 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-th74h pod in namespace openshift-dns (previous: false). I0512 03:00:37.567211 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-th74h pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-th74h\" is waiting to start: ContainerCreating" I0512 03:00:37.567230 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-th74h\" is waiting to start: ContainerCreating" I0512 03:00:37.567238 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-th74h pod in namespace openshift-dns (previous: false). I0512 03:00:37.756108 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-th74h pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-th74h\" is waiting to start: ContainerCreating" I0512 03:00:37.756126 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-th74h\" is waiting to start: ContainerCreating" I0512 03:00:37.756138 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-mnlbd pod in namespace openshift-dns (previous: false). W0512 03:00:37.935652 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0512 03:00:37.948698 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0512 03:00:37.948749 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-67596fb5f5-6ss7q pod in namespace openshift-image-registry (previous: false). I0512 03:00:38.142725 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0512 03:00:38.142762 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-67596fb5f5-mrcdm pod in namespace openshift-image-registry (previous: false). I0512 03:00:38.348798 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-67596fb5f5-mrcdm pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-67596fb5f5-mrcdm\" is waiting to start: ContainerCreating" I0512 03:00:38.348817 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-67596fb5f5-mrcdm\" is waiting to start: ContainerCreating" I0512 03:00:38.348861 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6b548c5c76-m64ws pod in namespace openshift-image-registry (previous: false). I0512 03:00:38.364316 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0512 03:00:38.542689 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0512 03:00:38.542710 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-5b94h pod in namespace openshift-image-registry (previous: false). I0512 03:00:38.747735 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0512 03:00:38.747754 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-548d975bbb-gl56p pod in namespace openshift-ingress (previous: false). I0512 03:00:38.767645 1 tasks_processing.go:74] worker 49 stopped. I0512 03:00:38.767695 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=dabc8a09b57f9f966249a4f500abadf0c784c9ba0a2dfc3d83d20b2cc6952cd8 I0512 03:00:38.767735 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=9321990725aa5bd2b23504b122d786b7a2a7367881fd73d48c91d30ba8d300f5 I0512 03:00:38.767765 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0512 03:00:38.767790 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=ca25a498e3277c4e13bba156f87569c3b038261ccf74676ef98d82af606805c3 I0512 03:00:38.767807 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0512 03:00:38.767830 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=e62f2cfeea03f5eaebbb617a4c89848fe6a7ec643dc3665636092657c691729f I0512 03:00:38.767860 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=83ebb65fb20989c254b8b1cd59f38dfee5fa7cdf634a990c55d78d4c16d760bd I0512 03:00:38.767953 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=a648c51523dcedb512a560dbbfff7e4ce2306cd72ea57b7f6a5170100c4d2e5c I0512 03:00:38.767970 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=e15785ea0482d1dff1b50d07ccdc08e8e1cf45abfec67627cfebfb9f130369cf I0512 03:00:38.767988 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=88a8f5a896422dd2afa49a8df8bf03fddf875870c0e208f9d599f651faaa6607 I0512 03:00:38.767998 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0512 03:00:38.768014 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=7fbe8aa8df22de333f3ab12e28b911ae5091b9c1d0b945eb6a9face1c4028297 I0512 03:00:38.768024 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0512 03:00:38.768039 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=434eade6a959d9f90b3b3f7ff844845d22fdcaa0bfb175b451e3f9a7015e51a8 I0512 03:00:38.768048 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0512 03:00:38.768062 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=04efbde4903af8a6c9934326afc2adfd26f976431a828ac893d977b8b5007134 I0512 03:00:38.768071 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0512 03:00:38.768086 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=92d40fa256f9c67b4f555edcb7f7105de5e09b8ecfe7d84a5deca72eb31a6047 I0512 03:00:38.768196 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=3b8fef07cc5feed5f5bc891a12e01cb8c8d7217b3dd4d96cfca1e098be4b45b0 I0512 03:00:38.768205 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0512 03:00:38.768213 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0512 03:00:38.768235 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0512 03:00:38.768258 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=bf5ca46b1a837c66dca95b3130807f40b913ff767a5603d52d0095de54d452e7 I0512 03:00:38.768281 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=af42582a81b010c33ead057c2bc036de3b54e65944ff6278ec819084fc1a7fc6 I0512 03:00:38.768290 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0512 03:00:38.768306 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=6da74f4396df74f1a00ea7bd1b8b017d20bb93af0537dacdfd70f2855963b2c3 I0512 03:00:38.768315 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0512 03:00:38.768329 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=549fc306f1669cac027bb972948d343e05d2ccc6ac8001975398e45de15f7da7 I0512 03:00:38.768344 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=c3383ae633140406787d0a71da98576e17e2a9b303e382c1e91cf6ec0c0ce7a1 I0512 03:00:38.768358 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=03c71af51ad115f0a56fd6aadc5e376e1df261271ee819627fcc3f174f565cc8 I0512 03:00:38.768373 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=b7212d6a0edd8cea25e242503bc5e8a639597f494d999598705538c0e2e6f511 I0512 03:00:38.768387 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=5941c0ef27c2b4d8d47b562b17320be38fd69553169dd0ad8dd4985b8c877f05 I0512 03:00:38.768414 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=1cca18e5a0a45e0b8b5f98e6230a7e6460e985d67e9f326d63aba23ae2a61e11 I0512 03:00:38.768430 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0512 03:00:38.768440 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0512 03:00:38.768446 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.880158524s to process 35 records W0512 03:00:38.935818 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0512 03:00:38.948290 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-548d975bbb-gl56p pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-548d975bbb-gl56p\" is waiting to start: ContainerCreating" I0512 03:00:38.948304 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-548d975bbb-gl56p\" is waiting to start: ContainerCreating" I0512 03:00:38.948314 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-548d975bbb-t5ggc pod in namespace openshift-ingress (previous: false). I0512 03:00:39.143340 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0512 03:00:39.143358 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5bdf876667-s6bb5 pod in namespace openshift-ingress (previous: false). I0512 03:00:39.349057 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5bdf876667-s6bb5 pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5bdf876667-s6bb5\" is waiting to start: ContainerCreating" I0512 03:00:39.349074 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5bdf876667-s6bb5\" is waiting to start: ContainerCreating" I0512 03:00:39.349106 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-htvcp pod in namespace openshift-ingress-canary (previous: false). I0512 03:00:39.549048 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-htvcp pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-htvcp\" is waiting to start: ContainerCreating" I0512 03:00:39.549065 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-htvcp\" is waiting to start: ContainerCreating" I0512 03:00:39.549083 1 tasks_processing.go:74] worker 33 stopped. I0512 03:00:39.549161 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=5d7d8e123680f2225d9525362934359d69e40e8b35262884ea652adc16d53767 I0512 03:00:39.549187 1 recorder.go:75] Recording events/openshift-dns with fingerprint=3e6b62073678db8c0a55fa158ceddcd095af3b946834e23d7cac6430d2e34d1b I0512 03:00:39.549253 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=96f2c9aab877bf4d1b05838268144f3faef4503dcbc7434378330066e1eb0cdf I0512 03:00:39.549280 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=658b6c59754a164c1a3057541af316027d5da518f26644576fb1f31cb091cfb9 I0512 03:00:39.549322 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=a7635bc1eb9d3050f942b6e62c3b1dfd4077f822a48557c2a1a05bcd85007dcc I0512 03:00:39.549332 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=220cd02e61278d4ac980dcf4d474b83930cdf5da66a36767dde795ca11890a2c I0512 03:00:39.550643 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-th74h with fingerprint=cb1b5bb0ed3d0802f49f88403ebce4da4c3fd43a45c97057bdc52c049ef12cb3 I0512 03:00:39.550740 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-67596fb5f5-6ss7q with fingerprint=0bca9f64806ac73f0889afeb23a513c340dd05cbabefb871179a2d7c970e0b44 I0512 03:00:39.550829 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-67596fb5f5-mrcdm with fingerprint=1c6c8a88f5aac08a682f7250d5acb3b09ddb7a2a0f1cdf1757507c91a42c9b31 I0512 03:00:39.550915 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-6b548c5c76-m64ws with fingerprint=947774ea1abab1efc830d45108bb6b7dd5f202c7dc9f1dc5b2055041106cad6a I0512 03:00:39.550972 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-htvcp with fingerprint=946c8eec457b19c91c68a7a0bc431bc23c6716ccbb8ee1207c087cc8858a06a3 I0512 03:00:39.550981 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 2.660555813s to process 11 records W0512 03:00:39.935234 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0512 03:00:40.935739 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0512 03:00:41.932651 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0512 03:00:41.932679 1 tasks_processing.go:74] worker 37 stopped. E0512 03:00:41.932689 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0512 03:00:41.932702 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0512 03:00:41.932719 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0512 03:00:41.932728 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.042551248s to process 1 records I0512 03:00:49.541491 1 tasks_processing.go:74] worker 36 stopped. I0512 03:00:49.541536 1 recorder.go:75] Recording config/installplans with fingerprint=95dfd5f33a9a46199239158a0ec0183b629818eca24ae41f4d706d7a3d604aec I0512 03:00:49.541549 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.653892094s to process 1 records I0512 03:00:49.969395 1 configmapobserver.go:84] configmaps "insights-config" not found I0512 03:00:50.132619 1 configmapobserver.go:84] configmaps "insights-config" not found I0512 03:00:50.170637 1 configmapobserver.go:84] configmaps "insights-config" not found I0512 03:00:50.297226 1 tasks_processing.go:74] worker 43 stopped. I0512 03:00:50.297482 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=a7a4de02df1cc1abdf9107bf21a9d837b0f58083251a7f44b02bf3cc6249ecc3 I0512 03:00:50.297498 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.409538648s to process 1 records E0512 03:00:50.297552 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.41s with: function \"machine_healthchecks\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0512 03:00:50.298651 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machine_healthchecks" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0512 03:00:50.298666 1 periodic.go:209] Running workloads gatherer I0512 03:00:50.298680 1 tasks_processing.go:45] number of workers: 2 I0512 03:00:50.298686 1 tasks_processing.go:69] worker 1 listening for tasks. I0512 03:00:50.298690 1 tasks_processing.go:71] worker 1 working on workload_info task. I0512 03:00:50.298755 1 tasks_processing.go:69] worker 0 listening for tasks. I0512 03:00:50.298830 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0512 03:00:50.334805 1 tasks_processing.go:74] worker 0 stopped. I0512 03:00:50.334822 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 35.957005ms to process 0 records I0512 03:00:50.414950 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 19s for image data I0512 03:00:50.426112 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (11ms) I0512 03:00:50.436641 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (11ms) I0512 03:00:50.449488 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (13ms) I0512 03:00:50.460009 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (10ms) I0512 03:00:50.472282 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (12ms) I0512 03:00:50.482627 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (10ms) I0512 03:00:50.492630 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (10ms) I0512 03:00:50.503011 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (10ms) I0512 03:00:50.513047 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (10ms) I0512 03:00:50.527172 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (14ms) I0512 03:00:50.537853 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (11ms) I0512 03:00:50.628057 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (90ms) I0512 03:00:50.727575 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (99ms) I0512 03:00:50.825817 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (98ms) I0512 03:00:50.926448 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (101ms) I0512 03:00:51.026848 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0512 03:00:51.127253 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0512 03:00:51.226820 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (100ms) I0512 03:00:51.327798 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (101ms) I0512 03:00:51.426756 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (99ms) I0512 03:00:51.526591 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (100ms) I0512 03:00:51.627136 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (101ms) I0512 03:00:51.726925 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0512 03:00:51.826531 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0512 03:00:51.926172 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0512 03:00:52.026678 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0512 03:00:52.126941 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0512 03:00:52.227044 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (100ms) I0512 03:00:52.326890 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (100ms) I0512 03:00:52.426865 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0512 03:00:52.526604 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (100ms) I0512 03:00:52.626363 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (100ms) I0512 03:00:52.626390 1 tasks_processing.go:74] worker 1 stopped. E0512 03:00:52.626400 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0512 03:00:52.626782 1 recorder.go:75] Recording config/workload_info with fingerprint=4ef0a2db8b7bd44c2963036b4cb89ac785c91fb9dabd86f35b4cb9c8d0ab3737 I0512 03:00:52.626799 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.327693715s to process 1 records E0512 03:00:52.626823 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.328s with: function \"workload_info\" failed with an error" I0512 03:00:52.627920 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0512 03:00:52.627932 1 periodic.go:209] Running conditional gatherer I0512 03:00:52.635521 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0512 03:00:52.641782 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.21:48542->172.30.0.10:53: read: connection refused E0512 03:00:52.642025 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0512 03:00:52.642081 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0512 03:00:52.649311 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0512 03:00:52.649324 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0512 03:00:52.649330 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0512 03:00:52.649333 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0512 03:00:52.649337 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0512 03:00:52.649340 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0512 03:00:52.649343 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0512 03:00:52.649348 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0512 03:00:52.649351 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0512 03:00:52.649354 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0512 03:00:52.649368 1 tasks_processing.go:45] number of workers: 3 I0512 03:00:52.649377 1 tasks_processing.go:69] worker 2 listening for tasks. I0512 03:00:52.649381 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0512 03:00:52.649386 1 tasks_processing.go:69] worker 0 listening for tasks. I0512 03:00:52.649396 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0512 03:00:52.649399 1 tasks_processing.go:69] worker 1 listening for tasks. I0512 03:00:52.649408 1 tasks_processing.go:74] worker 1 stopped. I0512 03:00:52.649401 1 tasks_processing.go:71] worker 0 working on conditional_gatherer_rules task. I0512 03:00:52.649469 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0512 03:00:52.649489 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.039µs to process 1 records I0512 03:00:52.649548 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0512 03:00:52.649556 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.048µs to process 1 records I0512 03:00:52.649563 1 tasks_processing.go:74] worker 0 stopped. I0512 03:00:52.649718 1 tasks_processing.go:74] worker 2 stopped. I0512 03:00:52.649731 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 329.626µs to process 0 records I0512 03:00:52.649750 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.21:48542->172.30.0.10:53: read: connection refused I0512 03:00:52.649767 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0512 03:00:52.676742 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=e5d86ff0327dc93df0b50b75683c8ea01bbc1883a8c931d29b417fccc67b7446 I0512 03:00:52.676856 1 diskrecorder.go:70] Writing 103 records to /var/lib/insights-operator/insights-2026-05-12-030052.tar.gz I0512 03:00:52.683036 1 diskrecorder.go:51] Wrote 103 records to disk in 6ms I0512 03:00:52.683064 1 periodic.go:278] Gathering cluster info every 2h0m0s I0512 03:00:52.683077 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0512 03:01:01.048039 1 configmapobserver.go:84] configmaps "insights-config" not found I0512 03:02:06.546328 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="0d394b858066519f6f40c77c47eaeca9e006b265a3630eb4dd7c89c4efbc927b") W0512 03:02:06.546363 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0512 03:02:06.546409 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="f5703bbb0f087782e4daa9734eb464f0b99a44bec3fb9962e03c6866d993f1b2") I0512 03:02:06.546411 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0512 03:02:06.546437 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0512 03:02:06.546475 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0512 03:02:06.546494 1 base_controller.go:181] Shutting down ConfigController ... I0512 03:02:06.546499 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="98084dfc3884792ad255d3af9c28efba4c9080baa8d6b72ea83d8663acffe549") I0512 03:02:06.546517 1 base_controller.go:181] Shutting down LoggingSyncer ... I0512 03:02:06.546524 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0512 03:02:06.546506 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0512 03:02:06.546535 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0512 03:02:06.546534 1 genericapiserver.go:651] "[graceful-termination] not going to wait for active watch request(s) to drain" I0512 03:02:06.546499 1 periodic.go:170] Shutting down