I0521 19:40:13.325454 1 cmd.go:253] Using service-serving-cert provided certificates I0521 19:40:13.325911 1 observer_polling.go:159] Starting file observer I0521 19:40:13.927643 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0521 19:40:13.927941 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0521 19:40:13.928832 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0521 19:40:13.928914 1 secure_serving.go:57] Forcing use of http/1.1 only W0521 19:40:13.928943 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0521 19:40:13.928952 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0521 19:40:13.928958 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0521 19:40:13.928965 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0521 19:40:13.928968 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0521 19:40:13.928972 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0521 19:40:13.933262 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0521 19:40:13.933564 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"94d15923-5983-4f8d-94ab-cb4eed32c4ab", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0521 19:40:13.935352 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0521 19:40:13.935371 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0521 19:40:13.935402 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0521 19:40:13.935424 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:40:13.935421 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0521 19:40:13.935436 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:40:13.935629 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0521 19:40:13.935952 1 secure_serving.go:213] Serving securely on [::]:8443 I0521 19:40:13.935985 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0521 19:40:13.939885 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0521 19:40:13.939913 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:40:13.940082 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0521 19:40:13.944709 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:40:13.944740 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0521 19:40:13.948877 1 secretconfigobserver.go:119] support secret does not exist I0521 19:40:13.953560 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:40:13.957253 1 secretconfigobserver.go:119] support secret does not exist I0521 19:40:13.960503 1 recorder.go:161] Pruning old reports every 7h27m30s, max age is 288h0m0s I0521 19:40:13.964818 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0521 19:40:13.964834 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0521 19:40:13.964834 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0521 19:40:13.964847 1 insightsreport.go:296] Starting report retriever I0521 19:40:13.964857 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0521 19:40:13.964884 1 periodic.go:209] Running clusterconfig gatherer I0521 19:40:13.964927 1 tasks_processing.go:45] number of workers: 64 I0521 19:40:13.964952 1 tasks_processing.go:69] worker 3 listening for tasks. I0521 19:40:13.964965 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:40:13.964965 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:40:13.964973 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:40:13.964977 1 tasks_processing.go:69] worker 22 listening for tasks. I0521 19:40:13.964983 1 tasks_processing.go:69] worker 42 listening for tasks. I0521 19:40:13.964985 1 tasks_processing.go:71] worker 22 working on storage_classes task. I0521 19:40:13.964988 1 tasks_processing.go:69] worker 17 listening for tasks. I0521 19:40:13.964992 1 tasks_processing.go:69] worker 5 listening for tasks. I0521 19:40:13.964994 1 tasks_processing.go:69] worker 33 listening for tasks. I0521 19:40:13.964990 1 tasks_processing.go:69] worker 43 listening for tasks. I0521 19:40:13.964985 1 tasks_processing.go:69] worker 4 listening for tasks. I0521 19:40:13.964999 1 tasks_processing.go:69] worker 44 listening for tasks. I0521 19:40:13.965005 1 tasks_processing.go:69] worker 16 listening for tasks. I0521 19:40:13.965000 1 tasks_processing.go:69] worker 13 listening for tasks. I0521 19:40:13.965001 1 tasks_processing.go:69] worker 34 listening for tasks. I0521 19:40:13.965002 1 tasks_processing.go:69] worker 32 listening for tasks. I0521 19:40:13.965013 1 tasks_processing.go:69] worker 23 listening for tasks. I0521 19:40:13.965006 1 tasks_processing.go:69] worker 45 listening for tasks. I0521 19:40:13.965021 1 tasks_processing.go:69] worker 47 listening for tasks. I0521 19:40:13.965020 1 tasks_processing.go:69] worker 15 listening for tasks. I0521 19:40:13.965008 1 tasks_processing.go:69] worker 35 listening for tasks. I0521 19:40:13.965013 1 tasks_processing.go:69] worker 46 listening for tasks. I0521 19:40:13.965037 1 tasks_processing.go:69] worker 11 listening for tasks. I0521 19:40:13.965014 1 tasks_processing.go:69] worker 6 listening for tasks. I0521 19:40:13.965044 1 tasks_processing.go:69] worker 24 listening for tasks. I0521 19:40:13.965046 1 tasks_processing.go:69] worker 14 listening for tasks. I0521 19:40:13.965050 1 tasks_processing.go:69] worker 37 listening for tasks. I0521 19:40:13.965015 1 tasks_processing.go:69] worker 36 listening for tasks. I0521 19:40:13.965021 1 tasks_processing.go:69] worker 9 listening for tasks. I0521 19:40:13.965021 1 tasks_processing.go:69] worker 7 listening for tasks. I0521 19:40:13.965028 1 tasks_processing.go:69] worker 10 listening for tasks. I0521 19:40:13.965032 1 tasks_processing.go:69] worker 48 listening for tasks. I0521 19:40:13.965064 1 tasks_processing.go:69] worker 41 listening for tasks. I0521 19:40:13.965030 1 tasks_processing.go:69] worker 28 listening for tasks. I0521 19:40:13.965032 1 tasks_processing.go:69] worker 27 listening for tasks. I0521 19:40:13.965040 1 tasks_processing.go:69] worker 49 listening for tasks. I0521 19:40:13.965047 1 tasks_processing.go:69] worker 50 listening for tasks. I0521 19:40:13.965053 1 tasks_processing.go:69] worker 25 listening for tasks. I0521 19:40:13.965375 1 tasks_processing.go:69] worker 8 listening for tasks. I0521 19:40:13.965384 1 tasks_processing.go:69] worker 26 listening for tasks. I0521 19:40:13.965060 1 tasks_processing.go:69] worker 40 listening for tasks. I0521 19:40:13.965390 1 tasks_processing.go:69] worker 29 listening for tasks. I0521 19:40:13.965394 1 tasks_processing.go:69] worker 51 listening for tasks. I0521 19:40:13.965347 1 tasks_processing.go:69] worker 20 listening for tasks. I0521 19:40:13.964978 1 tasks_processing.go:69] worker 12 listening for tasks. I0521 19:40:13.965434 1 tasks_processing.go:69] worker 18 listening for tasks. I0521 19:40:13.965447 1 tasks_processing.go:69] worker 38 listening for tasks. I0521 19:40:13.965472 1 tasks_processing.go:71] worker 38 working on qemu_kubevirt_launcher_logs task. I0521 19:40:13.965495 1 tasks_processing.go:69] worker 19 listening for tasks. I0521 19:40:13.965507 1 tasks_processing.go:71] worker 19 working on olm_operators task. I0521 19:40:13.965515 1 tasks_processing.go:71] worker 46 working on pod_network_connectivity_checks task. I0521 19:40:13.965525 1 tasks_processing.go:69] worker 59 listening for tasks. I0521 19:40:13.965528 1 tasks_processing.go:69] worker 31 listening for tasks. I0521 19:40:13.965542 1 tasks_processing.go:69] worker 52 listening for tasks. I0521 19:40:13.965551 1 tasks_processing.go:69] worker 53 listening for tasks. I0521 19:40:13.965547 1 tasks_processing.go:69] worker 39 listening for tasks. I0521 19:40:13.965562 1 tasks_processing.go:69] worker 30 listening for tasks. I0521 19:40:13.965566 1 tasks_processing.go:69] worker 21 listening for tasks. I0521 19:40:13.965566 1 tasks_processing.go:71] worker 2 working on machine_configs task. I0521 19:40:13.965545 1 tasks_processing.go:71] worker 16 working on storage_cluster task. I0521 19:40:13.965627 1 tasks_processing.go:69] worker 54 listening for tasks. I0521 19:40:13.965647 1 tasks_processing.go:71] worker 3 working on networks task. I0521 19:40:13.965655 1 tasks_processing.go:71] worker 23 working on schedulers task. I0521 19:40:13.965682 1 tasks_processing.go:69] worker 62 listening for tasks. I0521 19:40:13.965665 1 tasks_processing.go:71] worker 13 working on cluster_apiserver task. I0521 19:40:13.965666 1 tasks_processing.go:69] worker 61 listening for tasks. I0521 19:40:13.965666 1 tasks_processing.go:69] worker 63 listening for tasks. I0521 19:40:13.965671 1 tasks_processing.go:71] worker 34 working on container_runtime_configs task. I0521 19:40:13.965754 1 tasks_processing.go:71] worker 33 working on container_images task. I0521 19:40:13.965674 1 tasks_processing.go:69] worker 60 listening for tasks. I0521 19:40:13.965786 1 tasks_processing.go:71] worker 17 working on certificate_signing_requests task. I0521 19:40:13.965795 1 tasks_processing.go:71] worker 32 working on config_maps task. I0521 19:40:13.965840 1 tasks_processing.go:71] worker 43 working on dvo_metrics task. I0521 19:40:13.965842 1 tasks_processing.go:71] worker 5 working on authentication task. I0521 19:40:13.965860 1 tasks_processing.go:71] worker 4 working on ingress task. I0521 19:40:13.965889 1 tasks_processing.go:71] worker 11 working on aggregated_monitoring_cr_names task. I0521 19:40:13.965899 1 tasks_processing.go:71] worker 15 working on operators_pods_and_events task. I0521 19:40:13.965924 1 tasks_processing.go:71] worker 42 working on infrastructures task. I0521 19:40:13.966022 1 tasks_processing.go:69] worker 56 listening for tasks. I0521 19:40:13.966027 1 tasks_processing.go:71] worker 27 working on machine_config_pools task. I0521 19:40:13.966061 1 tasks_processing.go:71] worker 56 working on ceph_cluster task. I0521 19:40:13.966080 1 tasks_processing.go:71] worker 24 working on oauths task. I0521 19:40:13.966171 1 tasks_processing.go:71] worker 44 working on clusterroles task. I0521 19:40:13.964982 1 tasks_processing.go:71] worker 0 working on ingress_certificates task. I0521 19:40:13.966211 1 tasks_processing.go:71] worker 49 working on openstack_dataplanedeployments task. I0521 19:40:13.966289 1 tasks_processing.go:71] worker 14 working on openstack_controlplanes task. I0521 19:40:13.966394 1 tasks_processing.go:71] worker 59 working on lokistack task. I0521 19:40:13.966415 1 tasks_processing.go:69] worker 58 listening for tasks. I0521 19:40:13.966423 1 tasks_processing.go:71] worker 58 working on feature_gates task. I0521 19:40:13.966450 1 tasks_processing.go:71] worker 29 working on node_logs task. I0521 19:40:13.966853 1 tasks_processing.go:71] worker 10 working on openshift_logging task. I0521 19:40:13.966928 1 tasks_processing.go:71] worker 20 working on openshift_machine_api_events task. I0521 19:40:13.966064 1 tasks_processing.go:71] worker 6 working on nodenetworkconfigurationpolicies task. I0521 19:40:13.965799 1 tasks_processing.go:71] worker 47 working on proxies task. I0521 19:40:13.965934 1 tasks_processing.go:71] worker 45 working on validating_webhook_configurations task. I0521 19:40:13.966322 1 tasks_processing.go:69] worker 57 listening for tasks. I0521 19:40:13.965959 1 tasks_processing.go:69] worker 55 listening for tasks. I0521 19:40:13.966655 1 tasks_processing.go:71] worker 31 working on sap_config task. I0521 19:40:13.967644 1 tasks_processing.go:71] worker 55 working on crds task. I0521 19:40:13.966657 1 tasks_processing.go:71] worker 50 working on image_pruners task. I0521 19:40:13.966662 1 tasks_processing.go:71] worker 52 working on silenced_alerts task. I0521 19:40:13.966663 1 tasks_processing.go:71] worker 37 working on machine_autoscalers task. I0521 19:40:13.966668 1 tasks_processing.go:71] worker 53 working on sap_datahubs task. I0521 19:40:13.966673 1 tasks_processing.go:71] worker 39 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0521 19:40:13.966692 1 tasks_processing.go:71] worker 30 working on jaegers task. I0521 19:40:13.966696 1 tasks_processing.go:71] worker 21 working on overlapping_namespace_uids task. I0521 19:40:13.966702 1 tasks_processing.go:71] worker 54 working on nodes task. I0521 19:40:13.966725 1 tasks_processing.go:71] worker 62 working on install_plans task. I0521 19:40:13.966729 1 tasks_processing.go:71] worker 61 working on image task. I0521 19:40:13.966733 1 tasks_processing.go:71] worker 63 working on sap_pods task. I0521 19:40:13.966733 1 tasks_processing.go:71] worker 1 working on openstack_dataplanenodesets task. I0521 19:40:13.966737 1 tasks_processing.go:71] worker 60 working on machines task. I0521 19:40:13.966751 1 tasks_processing.go:71] worker 36 working on tsdb_status task. I0521 19:40:13.966753 1 tasks_processing.go:71] worker 25 working on pdbs task. I0521 19:40:13.966758 1 tasks_processing.go:71] worker 9 working on mutating_webhook_configurations task. I0521 19:40:13.966845 1 tasks_processing.go:71] worker 26 working on operators task. I0521 19:40:13.966852 1 tasks_processing.go:71] worker 40 working on openstack_version task. I0521 19:40:13.967015 1 tasks_processing.go:71] worker 51 working on service_accounts task. I0521 19:40:13.967052 1 tasks_processing.go:71] worker 48 working on cost_management_metrics_configs task. I0521 19:40:13.967043 1 tasks_processing.go:71] worker 8 working on image_registries task. I0521 19:40:13.967058 1 tasks_processing.go:71] worker 7 working on metrics task. I0521 19:40:13.967105 1 tasks_processing.go:71] worker 35 working on support_secret task. I0521 19:40:13.967123 1 tasks_processing.go:71] worker 41 working on version task. I0521 19:40:13.967131 1 tasks_processing.go:71] worker 28 working on active_alerts task. I0521 19:40:13.967162 1 tasks_processing.go:71] worker 12 working on nodenetworkstates task. I0521 19:40:13.967169 1 tasks_processing.go:71] worker 18 working on monitoring_persistent_volumes task. I0521 19:40:13.967638 1 tasks_processing.go:71] worker 57 working on machine_healthchecks task. E0521 19:40:13.973154 1 gather_active_alerts.go:64] Unable to retrieve most recent alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.128.0.20:53190->172.30.0.10:53: read: connection refused I0521 19:40:13.973170 1 tasks_processing.go:71] worker 28 working on machine_sets task. E0521 19:40:13.973206 1 gather.go:140] gatherer "clusterconfig" function "active_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.128.0.20:53190->172.30.0.10:53: read: connection refused I0521 19:40:13.973225 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 4.077583ms to process 0 records E0521 19:40:13.973225 1 gather_silenced_alerts.go:52] Unable to retrieve silenced alerts: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.128.0.20:53190->172.30.0.10:53: read: connection refused I0521 19:40:13.973244 1 tasks_processing.go:74] worker 52 stopped. E0521 19:40:13.973260 1 gather.go:140] gatherer "clusterconfig" function "silenced_alerts" failed with the error: Get "https://alertmanager-main.openshift-monitoring.svc:9094/api/v2/alerts?active=false&inhibited=false&silenced=true": dial tcp: lookup alertmanager-main.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.128.0.20:53190->172.30.0.10:53: read: connection refused I0521 19:40:13.973274 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 5.540612ms to process 0 records E0521 19:40:13.973297 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.128.0.20:40267->172.30.0.10:53: read: connection refused I0521 19:40:13.973312 1 tasks_processing.go:74] worker 36 stopped. E0521 19:40:13.973319 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.128.0.20:40267->172.30.0.10:53: read: connection refused I0521 19:40:13.973328 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 4.647171ms to process 0 records E0521 19:40:13.973420 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.128.0.20:40267->172.30.0.10:53: read: connection refused I0521 19:40:13.973440 1 tasks_processing.go:74] worker 7 stopped. E0521 19:40:13.973446 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.128.0.20:40267->172.30.0.10:53: read: connection refused I0521 19:40:13.973460 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 4.419934ms to process 0 records I0521 19:40:13.973523 1 tasks_processing.go:74] worker 16 stopped. I0521 19:40:13.973538 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 7.917526ms to process 0 records I0521 19:40:13.973687 1 tasks_processing.go:74] worker 23 stopped. I0521 19:40:13.973791 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=48d391ca925d82e76e60696a30527cf61da5297ddfe4435190a5084430056800 I0521 19:40:13.973805 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 7.990421ms to process 1 records I0521 19:40:13.973959 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0521 19:40:13.973979 1 controller.go:326] The initial operator extension status is healthy I0521 19:40:13.973994 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:40:13.974002 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:40:13.974007 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0521 19:40:13.974015 1 tasks_processing.go:74] worker 5 stopped. I0521 19:40:13.974024 1 controller.go:489] The operator is still being initialized I0521 19:40:13.974029 1 controller.go:512] The operator is healthy I0521 19:40:13.974149 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0521 19:40:13.974210 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0521 19:40:13.974233 1 recorder.go:75] Recording config/authentication with fingerprint=bb4c997ee40be08662c60e39756cd45584f223a65f2b7bc7f199b6df46584cbd I0521 19:40:13.974256 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 8.154858ms to process 1 records W0521 19:40:13.974287 1 operator.go:288] started I0521 19:40:13.974307 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0521 19:40:13.974374 1 tasks_processing.go:74] worker 4 stopped. I0521 19:40:13.974401 1 recorder.go:75] Recording config/ingress with fingerprint=42ed7061e94c43202e7db927aa22ac436d585e3a6fe3b923904d9775243b8b7d I0521 19:40:13.974412 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 8.179098ms to process 1 records I0521 19:40:13.975641 1 tasks_processing.go:74] worker 56 stopped. I0521 19:40:13.975653 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 9.562247ms to process 0 records I0521 19:40:13.975678 1 tasks_processing.go:74] worker 59 stopped. I0521 19:40:13.975690 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 9.266411ms to process 0 records I0521 19:40:13.982681 1 tasks_processing.go:74] worker 49 stopped. I0521 19:40:13.982694 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 16.456727ms to process 0 records I0521 19:40:13.983050 1 tasks_processing.go:74] worker 58 stopped. I0521 19:40:13.983151 1 recorder.go:75] Recording config/featuregate with fingerprint=6fbc14ece7c10b306eabb87ae6a54cf547121468a43715be25b01592ac9a85de I0521 19:40:13.983163 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 16.620278ms to process 1 records I0521 19:40:13.983244 1 tasks_processing.go:74] worker 50 stopped. I0521 19:40:13.983342 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=1c095ae5fcea1e51902a58e727e652845e5e3a87f22b3612b99d9fdf4dc8bb3e I0521 19:40:13.983350 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 15.372546ms to process 1 records I0521 19:40:13.983429 1 tasks_processing.go:74] worker 24 stopped. I0521 19:40:13.983563 1 recorder.go:75] Recording config/oauth with fingerprint=5fc0d8a67c3d0c0a0f77e80ed9b906e454b5d24a40280f1c7312dbeeb0d71705 I0521 19:40:13.983570 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 17.040255ms to process 1 records I0521 19:40:13.989060 1 tasks_processing.go:74] worker 27 stopped. I0521 19:40:13.989070 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 23.018262ms to process 0 records I0521 19:40:13.993017 1 tasks_processing.go:74] worker 31 stopped. I0521 19:40:13.993027 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 25.366879ms to process 0 records I0521 19:40:14.003997 1 tasks_processing.go:74] worker 53 stopped. I0521 19:40:14.004014 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 36.281161ms to process 0 records E0521 19:40:14.004024 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0521 19:40:14.004033 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 38.473833ms to process 0 records I0521 19:40:14.004040 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 37.142002ms to process 0 records I0521 19:40:14.004046 1 tasks_processing.go:74] worker 46 stopped. I0521 19:40:14.004053 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 36.329674ms to process 0 records I0521 19:40:14.004057 1 tasks_processing.go:74] worker 10 stopped. I0521 19:40:14.004058 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 37.695543ms to process 0 records I0521 19:40:14.004063 1 tasks_processing.go:74] worker 14 stopped. I0521 19:40:14.004067 1 tasks_processing.go:74] worker 37 stopped. I0521 19:40:14.004534 1 gather_logs.go:145] no pods in namespace were found I0521 19:40:14.004554 1 tasks_processing.go:74] worker 38 stopped. I0521 19:40:14.004566 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 39.069059ms to process 0 records I0521 19:40:14.004900 1 tasks_processing.go:74] worker 30 stopped. I0521 19:40:14.004963 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 37.154914ms to process 0 records I0521 19:40:14.004984 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 39.210264ms to process 0 records I0521 19:40:14.004994 1 tasks_processing.go:74] worker 34 stopped. I0521 19:40:14.005017 1 tasks_processing.go:74] worker 48 stopped. I0521 19:40:14.005046 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 36.03954ms to process 0 records I0521 19:40:14.005176 1 tasks_processing.go:74] worker 42 stopped. I0521 19:40:14.006802 1 recorder.go:75] Recording config/infrastructure with fingerprint=d060c4370b05a890305756f76a94a8a0f1db576d5191bdc1ab30d5db26827bbe I0521 19:40:14.006818 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 39.046362ms to process 1 records I0521 19:40:14.006863 1 recorder.go:75] Recording config/proxy with fingerprint=6b8cb3394488571b175dc7bae6d2b31067b221053e7aba91b695831b9f5f9379 I0521 19:40:14.006870 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 37.748528ms to process 1 records I0521 19:40:14.006896 1 tasks_processing.go:74] worker 47 stopped. I0521 19:40:14.006963 1 recorder.go:75] Recording config/network with fingerprint=5ef3ae55b2f024520979420d195d05c2a8b88a94b4a3db9d3d77f3b337cf9b40 I0521 19:40:14.006973 1 gather.go:177] gatherer "clusterconfig" function "networks" took 39.583831ms to process 1 records I0521 19:40:14.006961 1 tasks_processing.go:74] worker 3 stopped. I0521 19:40:14.007063 1 recorder.go:75] Recording config/apiserver with fingerprint=ebf12d479f451a3368bf879351698bfbfa3b45ee22a06a666fb189b371449763 I0521 19:40:14.007071 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 39.654554ms to process 1 records I0521 19:40:14.007077 1 tasks_processing.go:74] worker 13 stopped. I0521 19:40:14.011596 1 tasks_processing.go:74] worker 28 stopped. I0521 19:40:14.011609 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 38.418018ms to process 0 records I0521 19:40:14.011675 1 tasks_processing.go:74] worker 20 stopped. I0521 19:40:14.011701 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 44.731503ms to process 0 records I0521 19:40:14.012323 1 tasks_processing.go:74] worker 40 stopped. I0521 19:40:14.012340 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 43.453628ms to process 0 records I0521 19:40:14.012439 1 tasks_processing.go:74] worker 6 stopped. I0521 19:40:14.012451 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 45.121011ms to process 0 records I0521 19:40:14.012528 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=1c5b017119e0416d9a1c27ccb785233fd00c87723437836dd70aaf622c66b615 I0521 19:40:14.012532 1 tasks_processing.go:74] worker 22 stopped. I0521 19:40:14.012548 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=9d688908306d1d0a676249303b657f3f0c37d47e851e7d2f1a52d81850c7cff3 I0521 19:40:14.012557 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 47.455071ms to process 2 records I0521 19:40:14.012563 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 44.005288ms to process 0 records I0521 19:40:14.012567 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 44.071967ms to process 0 records I0521 19:40:14.012572 1 tasks_processing.go:74] worker 63 stopped. I0521 19:40:14.012574 1 tasks_processing.go:74] worker 1 stopped. I0521 19:40:14.012625 1 tasks_processing.go:74] worker 12 stopped. I0521 19:40:14.012639 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 43.505531ms to process 0 records I0521 19:40:14.012707 1 tasks_processing.go:74] worker 57 stopped. E0521 19:40:14.012736 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0521 19:40:14.012752 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 43.543734ms to process 0 records E0521 19:40:14.012761 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0521 19:40:14.012770 1 gather.go:177] gatherer "clusterconfig" function "machines" took 44.074234ms to process 0 records I0521 19:40:14.012778 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 45.874035ms to process 0 records I0521 19:40:14.012786 1 tasks_processing.go:74] worker 29 stopped. I0521 19:40:14.012809 1 tasks_processing.go:74] worker 60 stopped. I0521 19:40:14.012834 1 tasks_processing.go:74] worker 61 stopped. I0521 19:40:14.012947 1 recorder.go:75] Recording config/image with fingerprint=d10e5cb22c8de956196a7cb2555f92c6f310dbdbf1857414cc8c8be69771fccf I0521 19:40:14.012959 1 gather.go:177] gatherer "clusterconfig" function "image" took 44.450922ms to process 1 records I0521 19:40:14.013252 1 tasks_processing.go:74] worker 9 stopped. I0521 19:40:14.013417 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=60408e2b3228726854643075c87f260e2f08622c4f522913b73adc9bb7f0ecfa I0521 19:40:14.013445 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=76ee676961ba8bf977817e50d9843dc3e81a4f07b7f32b8b6d570a326f3ab3a7 I0521 19:40:14.013480 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=fd4ee098c3ca67b419d54fb592a9664f7862eee344f939d61add2cead6d23b5a I0521 19:40:14.013488 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 44.48274ms to process 3 records W0521 19:40:14.015335 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:40:14.019775 1 tasks_processing.go:74] worker 35 stopped. E0521 19:40:14.019794 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0521 19:40:14.019809 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 50.730403ms to process 0 records I0521 19:40:14.019851 1 tasks_processing.go:74] worker 25 stopped. I0521 19:40:14.019963 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=fde7a07e00456a5a3be0c8398c7406f0cd4a1610e11fe5237b43306463977ade I0521 19:40:14.019986 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=63e5ba6b8809b8bea5454755870d3f0e39c5d49a4dd7c5172ea9b31bf4b06aec I0521 19:40:14.020006 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=eee21e6a13cd088d7e53768f0521d2797faddae9da7d1d0086a652fe7c893608 I0521 19:40:14.020027 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=a13b861432315f7ad773b5af81e79c32d6e5bcaf795ef844b7e5c6c78e25b589 I0521 19:40:14.020047 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=cf3c6ee082b03997a463634964e2a654b654934dd2b5dd78eda207337a0c8239 I0521 19:40:14.020068 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=46d79a1a80371ff67b136a71a0192078fc4e2a71fa48db58d34eb86a817f8c3f I0521 19:40:14.020079 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 51.164242ms to process 6 records I0521 19:40:14.020901 1 tasks_processing.go:74] worker 18 stopped. I0521 19:40:14.020920 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 51.751893ms to process 0 records I0521 19:40:14.020988 1 tasks_processing.go:74] worker 45 stopped. I0521 19:40:14.021468 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=b8a9bcfb2496a41c9bdf6dc5e9ced6a01edf7af96d12d6ef99dfdbe8266a920f I0521 19:40:14.021707 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=e50d79b06ab7e76299408a4097016887c187b280c9f005a23affa80bf799a114 I0521 19:40:14.021815 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=c362e77571800fac28bca654d35a8243fc1d7e8a0a4acd0dbc0762fe3a496ea6 I0521 19:40:14.021899 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=82809d2a1c1ee263fb12806688445e63e79985c60b9c5a9d50a86a2e31275267 I0521 19:40:14.021979 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=f713dddec2ae85ba294f0cc8fbf7cbaa0250c70f8995b04f178c093e4f9b363c I0521 19:40:14.022049 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=d5ec54e2b43a32168eaacb6ebc3e09f875e6f8acd7322177a077d5475c11a4fb I0521 19:40:14.022116 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=cd3b126a95e957738589e35478d45bc429b86db255fc042e2d7a277dc7ac2cde I0521 19:40:14.022200 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=84449596071d2efdcfd3975394fdd543fcc33d78ee6a3f7441598a48965c4c2c I0521 19:40:14.022252 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=fc682e98e476cc2413657bb1c527455a1e8f71aa571476bb0db905b096aa0f8d I0521 19:40:14.022292 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=1e6cfe17b89c7baa0e3da0afb8896446f022519b9935868ff4f8aa59699c2599 I0521 19:40:14.022337 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=2cb169b10f9c4fd210e103981ef37217540584f8469cc400288a5099fab31b87 I0521 19:40:14.022351 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 53.39663ms to process 11 records I0521 19:40:14.023029 1 recorder.go:75] Recording config/node/ip-10-0-0-235.ec2.internal with fingerprint=f66ffe9d0212ac29404b411bdabbdba63c739cd8efa828c46e2615c60ed994df I0521 19:40:14.023149 1 recorder.go:75] Recording config/node/ip-10-0-1-129.ec2.internal with fingerprint=8f6150d210605023dab72810c5f3fdfdd55c796e0a9f1cb479b30b0638e630c9 I0521 19:40:14.023163 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 54.140636ms to process 2 records I0521 19:40:14.023239 1 recorder.go:75] Recording config/olm_operators with fingerprint=a060bd57f3facb6390e57d355d1afebba6422768a3cea3027c5dd8e5f4e7de9e I0521 19:40:14.023252 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 57.388904ms to process 1 records I0521 19:40:14.023257 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0521 19:40:14.023269 1 tasks_processing.go:74] worker 54 stopped. I0521 19:40:14.023272 1 controller.go:212] Source scaController *sca.Controller is not ready I0521 19:40:14.023280 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0521 19:40:14.023335 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0521 19:40:14.023344 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0521 19:40:14.023263 1 tasks_processing.go:74] worker 19 stopped. I0521 19:40:14.023414 1 controller.go:489] The operator is still being initialized I0521 19:40:14.023428 1 controller.go:512] The operator is healthy I0521 19:40:14.024011 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 58.013587ms to process 0 records I0521 19:40:14.024388 1 tasks_processing.go:74] worker 11 stopped. I0521 19:40:14.024520 1 tasks_processing.go:74] worker 17 stopped. I0521 19:40:14.024545 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 58.719496ms to process 0 records I0521 19:40:14.025333 1 tasks_processing.go:74] worker 8 stopped. I0521 19:40:14.026288 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=ab35b951b79e9fefac009c900b9d66457ba51ef3187924773188729968644f54 I0521 19:40:14.026304 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 56.323555ms to process 1 records I0521 19:40:14.026536 1 tasks_processing.go:74] worker 21 stopped. I0521 19:40:14.026562 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0521 19:40:14.026575 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 58.773856ms to process 1 records I0521 19:40:14.027317 1 tasks_processing.go:74] worker 44 stopped. I0521 19:40:14.027543 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=1bd43972a3f40a06ef9f0c7e19a0fcf89052123ce839b15b90d5eef2db3413ff I0521 19:40:14.027644 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=b905da39dfbad81b545d942d792bb663ca0dee61820b3f02a50bded53d1d3224 I0521 19:40:14.027655 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 61.10592ms to process 2 records I0521 19:40:14.027744 1 tasks_processing.go:74] worker 55 stopped. I0521 19:40:14.028216 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=cc712ce4e866c66da40086d9e326c4fc0c53c7adea53f3987fff279395a7ab8c I0521 19:40:14.028488 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=d471a1b604d45aa2515e982406460a6854551d7859c9c84451743272896406da I0521 19:40:14.028499 1 gather.go:177] gatherer "clusterconfig" function "crds" took 59.953861ms to process 2 records I0521 19:40:14.032611 1 tasks_processing.go:74] worker 2 stopped. I0521 19:40:14.032621 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0521 19:40:14.032636 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 66.725776ms to process 1 records E0521 19:40:14.034527 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%273197f273-0cad-4366-9ff4-8946f649d993%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.20:40028->172.30.0.10:53: read: connection refused I0521 19:40:14.034540 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%273197f273-0cad-4366-9ff4-8946f649d993%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.20:40028->172.30.0.10:53: read: connection refused I0521 19:40:14.035382 1 tasks_processing.go:74] worker 33 stopped. I0521 19:40:14.035429 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0521 19:40:14.035482 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0521 19:40:14.035506 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0521 19:40:14.036456 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0521 19:40:14.036700 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-zjcfn with fingerprint=f7b42f44de0cd7d8cbc0ecaa78ccbf7fb0a8ca2d0bcb4bb874dd4ddd8bb367e9 I0521 19:40:14.036780 1 recorder.go:75] Recording config/running_containers with fingerprint=8d020889ce732c865890a459c84cc8853e56955b4b8debc1fa62e62f8ee477fa I0521 19:40:14.036791 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 69.612976ms to process 2 records I0521 19:40:14.036867 1 tasks_processing.go:74] worker 39 stopped. I0521 19:40:14.036888 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 69.123455ms to process 0 records W0521 19:40:14.039735 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.20:42676->172.30.0.10:53: read: connection refused I0521 19:40:14.039747 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.20:42676->172.30.0.10:53: read: connection refused I0521 19:40:14.040138 1 base_controller.go:82] Caches are synced for ConfigController I0521 19:40:14.040151 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0521 19:40:14.044398 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:40:14.044900 1 tasks_processing.go:74] worker 0 stopped. E0521 19:40:14.044913 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0521 19:40:14.044919 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2qegnnglbapmtmq1bjs7srvhpig1kc2g-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2qegnnglbapmtmq1bjs7srvhpig1kc2g-primary-cert-bundle-secret" not found I0521 19:40:14.044961 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=9481fda5db892ac6afc1dfe42d7cc029c2366d62f13e68f5d281624cfc93e144 I0521 19:40:14.044973 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 78.706266ms to process 1 records I0521 19:40:14.048423 1 tasks_processing.go:74] worker 32 stopped. E0521 19:40:14.048435 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0521 19:40:14.048440 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0521 19:40:14.048444 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0521 19:40:14.048452 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0521 19:40:14.048475 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0521 19:40:14.048482 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0521 19:40:14.048487 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0521 19:40:14.048495 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0521 19:40:14.048532 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0521 19:40:14.048539 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0521 19:40:14.048544 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 82.608514ms to process 7 records I0521 19:40:14.075036 1 base_controller.go:82] Caches are synced for LoggingSyncer I0521 19:40:14.075048 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0521 19:40:14.084683 1 tasks_processing.go:74] worker 41 stopped. I0521 19:40:14.085085 1 recorder.go:75] Recording config/version with fingerprint=c62bd0b4aff7acfb02ea7c1a173d463ef1aa858a177e84de582a9ee3213f0a70 I0521 19:40:14.085100 1 recorder.go:75] Recording config/id with fingerprint=c166e70f25e771b7e1eb4b9a408ee5a7901f2442956f42668a41ba8e974bb1b7 I0521 19:40:14.085106 1 gather.go:177] gatherer "clusterconfig" function "version" took 115.613238ms to process 2 records I0521 19:40:14.087342 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0521 19:40:14.091741 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found W0521 19:40:15.015590 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:40:15.634530 1 gather_cluster_operator_pods_and_events.go:121] Found 20 pods with 21 containers I0521 19:40:15.634544 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1198372 bytes I0521 19:40:15.635737 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for networking-console-plugin container networking-console-plugin-6ddbfdf749-xfxqb pod in namespace openshift-network-console (previous: false). I0521 19:40:15.845855 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0521 19:40:15.861988 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for networking-console-plugin container networking-console-plugin-6ddbfdf749-xpmzc pod in namespace openshift-network-console (previous: false). W0521 19:40:16.015117 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:40:16.041331 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console-operator container console-operator-575cd97545-zjcfn pod in namespace openshift-console-operator (previous: true). I0521 19:40:16.236147 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console-operator container console-operator-575cd97545-zjcfn pod in namespace openshift-console-operator (previous: false). I0521 19:40:16.441398 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console container console-6b686dd4fd-b8826 pod in namespace openshift-console (previous: false). I0521 19:40:16.447572 1 tasks_processing.go:74] worker 26 stopped. I0521 19:40:16.447691 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=5aa7fe3048ce81fd4c62906be3bdac25bd71bbb309ad88182ead0b1c7e02d6aa I0521 19:40:16.447762 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=b1d3a57201f472fba248e6e22732d03f4ebac77f3d7204455295e9c6f31f376a I0521 19:40:16.447808 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=b5186982179455990f5ac46f7c5a84ddec7b4626b4b4d1a0426a7e21ee3ff4b2 I0521 19:40:16.447825 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0521 19:40:16.447851 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=e281bdaafbfdb745ca7fe945e3081d415a3c21406ade8ac71c55ad58e1324ce1 I0521 19:40:16.447867 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0521 19:40:16.447888 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=df10824319f8c2be3b8a38b5ce534408b953b966f83d5ce073fc8fd25f8cd9d1 I0521 19:40:16.447916 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=eea947bdf4e6ff475fcd61ded52752618d869ca272d8d07b61ded498739d6361 I0521 19:40:16.447939 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=5636ffc2efc15b5cddb1590c558323f9b8cdf874a04d60fac0e2e777d7c4c619 I0521 19:40:16.447968 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=c91a351af97217fa58611173b86d0b232112b2228fcf1b72e8982957a710e46c I0521 19:40:16.448413 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=3fde2d11033d115ab0157e4020a04ae53cb9db35c6b4dc15233fa573dfda367a I0521 19:40:16.448483 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=75fff9c55b0a1811b4ffdc2d239abeabdafbb510c681454ecc160ead627044b6 I0521 19:40:16.448508 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0521 19:40:16.448539 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=4e5cd7f860ec263514b183b3c4630c1c3c79ea2b2863874efc45e8ee304cdd26 I0521 19:40:16.448555 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0521 19:40:16.448584 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=08380563d89abfff27b2a420831d7da506380227f32a581d6fe48ed13610c0c5 I0521 19:40:16.448597 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0521 19:40:16.448628 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=a38b7fbb3f6acbd364e6714adb61ab4694dce1a5e6a666760548205f3c10497a I0521 19:40:16.448641 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0521 19:40:16.448677 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=72ff0a6ee44a93f584193bebcfe224a2afb5ce335efb69f24adbb51fb5f0ded1 I0521 19:40:16.448808 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=146134074a8c813e62a91d8322c55d3607e7a47f885d2d70d47fc8169b4ebbe1 I0521 19:40:16.448821 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0521 19:40:16.448828 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0521 19:40:16.448853 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0521 19:40:16.448876 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=fdaf442298e63baddb036289931b1ea7d2145b74ca8d0369566c283cc4a28df0 I0521 19:40:16.448899 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=418612f5a6132c874d636707d896666d5ac5ccc1c267c2e0fc887dc5b53b73ae I0521 19:40:16.448908 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0521 19:40:16.448924 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=d6f0682f2ec0b0271077de4ad67f64496b200b258e4cd9a99746748dea0366ec I0521 19:40:16.448934 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0521 19:40:16.448947 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=39e1a6383757b6a159a004d1c58f2f207144c54a64cc15d79f2e748b207376c8 I0521 19:40:16.448962 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=73431bb7bd4b342b3b6937bf43ead2380f009d2f0f724d60f6754e4f939e7774 I0521 19:40:16.448976 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=523ffc709bdfff72a773298f3f4a7b25ce40e6566b5d293dadd2d8b82139f7e9 I0521 19:40:16.448990 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=4f83c72e50269dc5e30b0ef8088c91a2577d69b3511fd95b228d8be4877b5bc8 I0521 19:40:16.449011 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=bc5d5352cbdea1426b951b6c2ac305675fc60a9a301de7f848117f1339304250 I0521 19:40:16.449020 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0521 19:40:16.449047 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=b63f0330d62896fbcc2b6ae7a9f0be47100417b25af9acd256ed07d959b61f10 I0521 19:40:16.449062 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0521 19:40:16.449069 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0521 19:40:16.449076 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.478735592s to process 38 records I0521 19:40:16.636447 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console container console-6c9c9456b4-kbz9t pod in namespace openshift-console (previous: false). I0521 19:40:16.833804 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:40:16.833821 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console container console-6c9c9456b4-xrpsl pod in namespace openshift-console (previous: false). W0521 19:40:17.015154 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:40:17.031872 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:40:17.031884 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for download-server container downloads-56cb7c9577-9v4jg pod in namespace openshift-console (previous: false). I0521 19:40:17.238955 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for download-server container downloads-56cb7c9577-wn642 pod in namespace openshift-console (previous: false). I0521 19:40:17.431415 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:40:17.431448 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-hgsfx pod in namespace openshift-dns (previous: false). I0521 19:40:17.650838 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-hgsfx pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-hgsfx\" is waiting to start: ContainerCreating" I0521 19:40:17.650864 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-hgsfx\" is waiting to start: ContainerCreating" I0521 19:40:17.650872 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-hgsfx pod in namespace openshift-dns (previous: false). I0521 19:40:17.855817 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-hgsfx pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-hgsfx\" is waiting to start: ContainerCreating" I0521 19:40:17.855835 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-hgsfx\" is waiting to start: ContainerCreating" I0521 19:40:17.855844 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-4wds2 pod in namespace openshift-dns (previous: false). W0521 19:40:18.015679 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0521 19:40:18.041149 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:40:18.041164 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-zl2qv pod in namespace openshift-dns (previous: false). I0521 19:40:18.260350 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:40:18.260391 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-599bcb9548-vz54r pod in namespace openshift-image-registry (previous: true). I0521 19:40:18.440254 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-599bcb9548-vz54r pod in namespace openshift-image-registry (previous: false). I0521 19:40:18.637995 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-68c9bd69ff-cdz9x pod in namespace openshift-image-registry (previous: false). I0521 19:40:18.833586 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:40:18.833603 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-68c9bd69ff-cwtpz pod in namespace openshift-image-registry (previous: false). W0521 19:40:19.011693 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:40:19.011745 1 tasks_processing.go:74] worker 43 stopped. E0521 19:40:19.011758 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0521 19:40:19.011770 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0521 19:40:19.011780 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.045890133s to process 1 records I0521 19:40:19.032108 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0521 19:40:19.032121 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-8hs7d pod in namespace openshift-image-registry (previous: false). I0521 19:40:19.240508 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-vc6jm pod in namespace openshift-image-registry (previous: false). I0521 19:40:19.445145 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5745dbb8dc-v698z pod in namespace openshift-ingress (previous: false). I0521 19:40:19.639766 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7fdccd76d7-c9zjk pod in namespace openshift-ingress (previous: false). I0521 19:40:19.838352 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-7fdccd76d7-c9zjk pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-7fdccd76d7-c9zjk\" is waiting to start: ContainerCreating" I0521 19:40:19.838369 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-7fdccd76d7-c9zjk\" is waiting to start: ContainerCreating" I0521 19:40:19.838381 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-7fdccd76d7-zvlzl pod in namespace openshift-ingress (previous: false). I0521 19:40:20.039296 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-t9d75 pod in namespace openshift-ingress-canary (previous: false). I0521 19:40:20.238526 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-t9d75 pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-t9d75\" is waiting to start: ContainerCreating" I0521 19:40:20.238543 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-t9d75\" is waiting to start: ContainerCreating" I0521 19:40:20.238562 1 tasks_processing.go:74] worker 15 stopped. I0521 19:40:20.238654 1 recorder.go:75] Recording events/openshift-network-console with fingerprint=6642dcb1498664172f49769d5b2dadeb5d61e37cfcf774c932d13628444c29cf I0521 19:40:20.238807 1 recorder.go:75] Recording events/openshift-console-operator with fingerprint=8294a54c77d9a5f25a793880b26feb56bd574b0b37a431b41ac2e5f31eef0c72 I0521 19:40:20.238858 1 recorder.go:75] Recording events/openshift-console with fingerprint=8bc1796b422d1ee19a0bced2284d9c52d884bffaa43709dfc88bb452c81011b6 I0521 19:40:20.238897 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=d3e79c488d9eec439d1857921b1421976ea65060bfda5a64900d984078bdb637 I0521 19:40:20.238923 1 recorder.go:75] Recording events/openshift-dns with fingerprint=59ccb12c4c2320cd153d0e9ccfc44fbe7f32baf2101d779b9e5a9f07cf671be8 I0521 19:40:20.239026 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=d2a7e2cc15b9704c128b1f2e53fd806861845bd541e8493509232973e71f4c02 I0521 19:40:20.239052 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=e6f836baf42604f22834ad1b1507a02efc0b548014d64cdbac92981949421fe2 I0521 19:40:20.239113 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=64bf5f4f8ca222584c2341f0644dad3f1946c1a43afbd5850b6c7e73a4dcb49e I0521 19:40:20.239124 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=50c98d789196aeefadd1f5bb04fa07b6372cf4cf6eb4d3f675d30da55ef4aec9 I0521 19:40:20.239131 1 recorder.go:75] Recording config/pod/openshift-network-console/logs/networking-console-plugin-6ddbfdf749-xfxqb/networking-console-plugin_current.log with fingerprint=4f05517563f0a2311136315dcdf2d0281a6d90defeefae9fc2bd6608fc89c656 I0521 19:40:20.239138 1 recorder.go:75] Recording config/pod/openshift-network-console/logs/networking-console-plugin-6ddbfdf749-xpmzc/networking-console-plugin_current.log with fingerprint=648f57dcff6f50edaa2e2820e3e0ecda0bc6aedd8a9269c0b6de85d55cac57b6 I0521 19:40:20.239270 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-zjcfn with fingerprint=f7b42f44de0cd7d8cbc0ecaa78ccbf7fb0a8ca2d0bcb4bb874dd4ddd8bb367e9 E0521 19:40:20.239285 1 gather.go:161] error recording gatherer "clusterconfig" function "operators_pods_and_events" result "config/pod/openshift-console-operator/console-operator-575cd97545-zjcfn.json" because of the error: the record with the same name "config/pod/openshift-console-operator/console-operator-575cd97545-zjcfn.json" was already recorded and had the fingerprint "f7b42f44de0cd7d8cbc0ecaa78ccbf7fb0a8ca2d0bcb4bb874dd4ddd8bb367e9", overwriting with the record having fingerprint "f7b42f44de0cd7d8cbc0ecaa78ccbf7fb0a8ca2d0bcb4bb874dd4ddd8bb367e9" W0521 19:40:20.239296 1 gather.go:155] issue recording gatherer "clusterconfig" function "operators_pods_and_events" result "config/pod/openshift-console-operator/console-operator-575cd97545-zjcfn.json" because of the warning: warning: the record with the same fingerprint "f7b42f44de0cd7d8cbc0ecaa78ccbf7fb0a8ca2d0bcb4bb874dd4ddd8bb367e9" was already recorded at path "config/pod/openshift-console-operator/console-operator-575cd97545-zjcfn.json", recording another one with a different path "config/pod/openshift-console-operator/console-operator-575cd97545-zjcfn.json" I0521 19:40:20.239307 1 recorder.go:75] Recording config/pod/openshift-console-operator/logs/console-operator-575cd97545-zjcfn/console-operator_previous.log with fingerprint=7d9bf9d706627da0ad169258ae1fa73c22f580a9464e2449a30f7b725ef1bb61 I0521 19:40:20.239476 1 recorder.go:75] Recording config/pod/openshift-console-operator/logs/console-operator-575cd97545-zjcfn/console-operator_current.log with fingerprint=cb96e1bfbda1ef30a1206f624ae235505a393d00b8b43c339789699c37c968c1 I0521 19:40:20.239489 1 recorder.go:75] Recording config/pod/openshift-console/logs/console-6b686dd4fd-b8826/console_current.log with fingerprint=be9bbb0d6e0e8c1a2cb8a2b9fb55e96f86e27c912648573d1687119263138111 I0521 19:40:20.239495 1 recorder.go:75] Recording config/pod/openshift-console/logs/downloads-56cb7c9577-9v4jg/download-server_current.log with fingerprint=b24143ce133fe8f0a3eeb853781a917c0a86728bfe5f4ae738fea8d1683d1d56 I0521 19:40:20.239587 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-hgsfx with fingerprint=2ab9ddfaff8f54174cafde057aa3e9f3b67074ec21a02c21031d848eb50054c4 I0521 19:40:20.239698 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-599bcb9548-vz54r with fingerprint=b181bb07bf8cf8965260bc8c45e02c5c20ec114a566674bf580f268f11e9f6b6 I0521 19:40:20.239744 1 recorder.go:75] Recording config/pod/openshift-image-registry/logs/image-registry-599bcb9548-vz54r/registry_previous.log with fingerprint=9812da4e4c9389f1dbc4d8c8fdbdd55f2a766f1ee45838fe4bd8574e8f18d587 I0521 19:40:20.239757 1 recorder.go:75] Recording config/pod/openshift-image-registry/logs/image-registry-599bcb9548-vz54r/registry_current.log with fingerprint=36e9dddb630d5c224dbdc03da87b0c20aaf70eb26850b4e42dd62924978ff6da I0521 19:40:20.239786 1 recorder.go:75] Recording config/pod/openshift-image-registry/logs/node-ca-8hs7d/node-ca_current.log with fingerprint=98eb2ab90fc37bf52ecb420e9a25fc81e4fe947be03fd8d9114bc37506ca76b2 I0521 19:40:20.239793 1 recorder.go:75] Recording config/pod/openshift-image-registry/logs/node-ca-vc6jm/node-ca_current.log with fingerprint=5c9b2b8d59ce28e41bc7a1876d533fa5334b7253e6f8195d52b633f2e09301e3 I0521 19:40:20.239813 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-5745dbb8dc-v698z/router_current.log with fingerprint=5d8fa7e6bc303acdccde9f8594018581cb96fb0fd5380facb1c6c29e9ee8eb66 I0521 19:40:20.239831 1 recorder.go:75] Recording config/pod/openshift-ingress/logs/router-default-7fdccd76d7-zvlzl/router_current.log with fingerprint=805e6518b56ffd9c2958959e99933c9bd992380b3587de4951d1483a3d3df4ec I0521 19:40:20.239889 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/ingress-canary-t9d75 with fingerprint=8d15725d15d6e86bdec1a5a6de8bd2e40bc0e14a02acc5020a0fc3f130d4f395 I0521 19:40:20.239897 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 6.272648774s to process 25 records I0521 19:40:27.032388 1 tasks_processing.go:74] worker 62 stopped. I0521 19:40:27.032432 1 recorder.go:75] Recording config/installplans with fingerprint=95dfd5f33a9a46199239158a0ec0183b629818eca24ae41f4d706d7a3d604aec I0521 19:40:27.032444 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 13.064213505s to process 1 records I0521 19:40:27.775359 1 tasks_processing.go:74] worker 51 stopped. I0521 19:40:27.775633 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=bae76059ad12f92c03c06b36414d832a0dbbce27d94c635a80399044861a2efd I0521 19:40:27.775650 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.80644495s to process 1 records E0521 19:40:27.775730 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.81s with: function \"active_alerts\" failed with an error, function \"silenced_alerts\" failed with an error, function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error, unable to record function \"operators_pods_and_events\" record \"config/pod/openshift-console-operator/console-operator-575cd97545-zjcfn.json\"" I0521 19:40:27.776835 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "active_alerts" failed with an error, function "silenced_alerts" failed with an error, function "tsdb_status" failed with an error, function "metrics" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error, unable to record function "operators_pods_and_events" record "config/pod/openshift-console-operator/console-operator-575cd97545-zjcfn.json" I0521 19:40:27.776848 1 periodic.go:209] Running workloads gatherer I0521 19:40:27.776864 1 tasks_processing.go:45] number of workers: 2 I0521 19:40:27.776873 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:40:27.776877 1 tasks_processing.go:71] worker 1 working on workload_info task. I0521 19:40:27.776881 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:40:27.776891 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0521 19:40:27.782656 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-129.ec2.internal... I0521 19:40:27.801763 1 tasks_processing.go:74] worker 0 stopped. I0521 19:40:27.801780 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 24.859807ms to process 0 records I0521 19:40:31.274488 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 3s I0521 19:40:31.292354 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 21s for image data I0521 19:40:31.301552 1 gather_workloads_info.go:387] No image sha256:80748ba08e1c264a8c105e7f607eff386a66378e024443a844993ee9292858c1 (10ms) I0521 19:40:31.310425 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (9ms) I0521 19:40:31.319466 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (9ms) I0521 19:40:31.329098 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (10ms) I0521 19:40:31.337770 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (9ms) I0521 19:40:31.347475 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (10ms) I0521 19:40:31.356285 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (9ms) I0521 19:40:31.365137 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (9ms) I0521 19:40:31.373992 1 gather_workloads_info.go:387] No image sha256:f995093d6fd6c66e2c50cc4bceee9c739d09453ea85eca0232013632371fc594 (9ms) I0521 19:40:31.382139 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (8ms) I0521 19:40:31.401314 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (19ms) I0521 19:40:31.501144 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0521 19:40:31.601595 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (100ms) I0521 19:40:31.701756 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0521 19:40:31.801869 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0521 19:40:31.903677 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (102ms) I0521 19:40:32.001456 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (98ms) I0521 19:40:32.102459 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (101ms) I0521 19:40:32.201948 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (99ms) I0521 19:40:32.303936 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (102ms) I0521 19:40:32.401444 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (97ms) I0521 19:40:32.501299 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0521 19:40:32.602017 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (101ms) I0521 19:40:32.701111 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (99ms) I0521 19:40:32.804658 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (104ms) I0521 19:40:32.901094 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (96ms) I0521 19:40:33.001658 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (101ms) I0521 19:40:33.100850 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (99ms) I0521 19:40:33.201078 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (100ms) I0521 19:40:33.301505 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (100ms) I0521 19:40:33.400808 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (99ms) I0521 19:40:33.501604 1 gather_workloads_info.go:387] No image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (101ms) I0521 19:40:33.601795 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (100ms) I0521 19:40:33.702113 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0521 19:40:33.801930 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (100ms) I0521 19:40:33.901279 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (99ms) I0521 19:40:34.001486 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (100ms) I0521 19:40:34.101710 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (100ms) I0521 19:40:34.202979 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (101ms) I0521 19:40:34.203008 1 tasks_processing.go:74] worker 1 stopped. I0521 19:40:34.203235 1 recorder.go:75] Recording config/workload_info with fingerprint=2c66fdd8301a245b2e8fdab51abf140b137f25bafe8dc480e30951b4394fc45f I0521 19:40:34.203250 1 gather.go:177] gatherer "workloads" function "workload_info" took 6.42612416s to process 1 records I0521 19:40:34.203265 1 periodic.go:256] Periodic gather workloads completed in 6.426s I0521 19:40:34.203275 1 controllerstatus.go:80] name=periodic-workloads healthy=true reason= message= I0521 19:40:34.203279 1 periodic.go:209] Running conditional gatherer I0521 19:40:34.209471 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0521 19:40:34.217554 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.20:33990->172.30.0.10:53: read: connection refused I0521 19:40:34.217833 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer E0521 19:40:34.221185 1 conditional_gatherer.go:324] unable to update alerts cache: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?match%5B%5D=ALERTS%7Balertstate%3D%22firing%22%7D&query=ALERTS": dial tcp: lookup prometheus-k8s.openshift-monitoring.svc on 172.30.0.10:53: read udp 10.128.0.20:52221->172.30.0.10:53: read: connection refused I0521 19:40:34.221263 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0521 19:40:34.227214 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0521 19:40:34.227230 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:40:34.227235 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:40:34.227239 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:40:34.227242 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:40:34.227245 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:40:34.227248 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:40:34.227251 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:40:34.227254 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0521 19:40:34.227257 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0521 19:40:34.227271 1 tasks_processing.go:45] number of workers: 3 I0521 19:40:34.227279 1 tasks_processing.go:69] worker 2 listening for tasks. I0521 19:40:34.227283 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0521 19:40:34.227290 1 tasks_processing.go:69] worker 0 listening for tasks. I0521 19:40:34.227301 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0521 19:40:34.227305 1 tasks_processing.go:69] worker 1 listening for tasks. I0521 19:40:34.227321 1 tasks_processing.go:74] worker 1 stopped. I0521 19:40:34.227320 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0521 19:40:34.227358 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0521 19:40:34.227370 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 737ns to process 1 records I0521 19:40:34.227400 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0521 19:40:34.227408 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.182µs to process 1 records I0521 19:40:34.227413 1 tasks_processing.go:74] worker 0 stopped. I0521 19:40:34.227590 1 tasks_processing.go:74] worker 2 stopped. I0521 19:40:34.227603 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 253.282µs to process 0 records I0521 19:40:34.227622 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.20:33990->172.30.0.10:53: read: connection refused I0521 19:40:34.227639 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0521 19:40:34.255204 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=95e6cf859d8f5f2f62fde14ba1f70bdb905702dd94f966e0058faa5e75d104de I0521 19:40:34.255359 1 diskrecorder.go:70] Writing 124 records to /var/lib/insights-operator/insights-2026-05-21-194034.tar.gz I0521 19:40:34.265037 1 diskrecorder.go:51] Wrote 124 records to disk in 9ms I0521 19:40:34.265065 1 periodic.go:278] Gathering cluster info every 2h0m0s I0521 19:40:34.265079 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0521 19:41:43.965372 1 diskrecorder.go:223] Found files to send: insights-2026-05-21-194034.tar.gz I0521 19:41:43.965407 1 insightsuploader.go:150] Checking archives to upload periodically every 15m5.45832476s I0521 19:41:43.965415 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0521 19:41:43.975733 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0521 19:41:55.230192 1 requests.go:69] gateway server https://console.redhat.com/api/ingress/v1/upload returned 401, x-rh-insights-request-id=cb4eedb724924b25bd573b606c617cdb I0521 19:41:55.230353 1 insightsuploader.go:169] Unable to upload report after 11.26s: your Red Hat account is not enabled for remote support or your token has expired: {"errors":[{"status":401,"detail":"UHC services authentication failed","meta":{"response_by":"gateway"}}]} I0521 19:41:55.230377 1 controllerstatus.go:80] name=insightsuploader healthy=false reason=NotAuthorized message=Reporting was not allowed: your Red Hat account is not enabled for remote support or your token has expired: {"errors":[{"status":401,"detail":"UHC services authentication failed","meta":{"response_by":"gateway"}}]} I0521 19:42:13.981095 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:42:13.981134 1 controller.go:512] The operator is healthy I0521 19:44:13.994731 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:44:13.994769 1 controller.go:512] The operator is healthy I0521 19:45:13.958090 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 19:45:13.967144 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:45:13.967162 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 19:45:13.971286 1 secretconfigobserver.go:119] support secret does not exist I0521 19:46:13.983378 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:46:13.983432 1 controller.go:512] The operator is healthy I0521 19:48:13.980893 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:48:13.980935 1 controller.go:512] The operator is healthy I0521 19:50:13.957706 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:50:13.961837 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 19:50:13.971935 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 19:50:13.977744 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:50:13.977768 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 19:50:13.981802 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:50:13.981833 1 controller.go:512] The operator is healthy I0521 19:50:13.982450 1 secretconfigobserver.go:119] support secret does not exist I0521 19:52:14.083393 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:52:14.083435 1 controller.go:512] The operator is healthy I0521 19:54:13.981226 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:54:13.981267 1 controller.go:512] The operator is healthy I0521 19:55:13.982832 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 19:55:13.988681 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 19:55:13.988702 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 19:55:13.993085 1 secretconfigobserver.go:119] support secret does not exist I0521 19:56:13.981143 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:56:13.981184 1 controller.go:512] The operator is healthy I0521 19:58:13.980900 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 19:58:13.980937 1 controller.go:512] The operator is healthy I0521 20:00:13.958953 1 configmapobserver.go:84] configmaps "insights-config" not found I0521 20:00:13.980999 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 20:00:13.981038 1 controller.go:512] The operator is healthy I0521 20:00:13.993817 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 20:00:13.999160 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 20:00:13.999174 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 20:00:14.003319 1 secretconfigobserver.go:119] support secret does not exist I0521 20:02:13.985374 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 20:02:13.985413 1 controller.go:512] The operator is healthy I0521 20:04:13.985089 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 20:04:13.985129 1 controller.go:512] The operator is healthy I0521 20:05:14.004380 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0521 20:05:14.010047 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0521 20:05:14.010066 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0521 20:05:14.014266 1 secretconfigobserver.go:119] support secret does not exist I0521 20:06:13.984385 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 20:06:13.984423 1 controller.go:512] The operator is healthy I0521 20:08:13.984844 1 controller.go:229] Number of last upload failures 1 lower than threshold 5. Not marking as degraded. I0521 20:08:13.984888 1 controller.go:512] The operator is healthy