I0505 23:20:40.492259 1 cmd.go:253] Using service-serving-cert provided certificates I0505 23:20:40.492691 1 observer_polling.go:159] Starting file observer I0505 23:20:41.189640 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0505 23:20:41.189848 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0505 23:20:41.190422 1 secure_serving.go:57] Forcing use of http/1.1 only I0505 23:20:41.190433 1 simple_featuregate_reader.go:171] Starting feature-gate-detector W0505 23:20:41.190446 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0505 23:20:41.190451 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0505 23:20:41.190455 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0505 23:20:41.190458 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0505 23:20:41.190461 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0505 23:20:41.190464 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0505 23:20:41.195552 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0505 23:20:41.196028 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"cecc487e-ed25-44ce-860d-de1b10325953", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0505 23:20:41.197164 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0505 23:20:41.197167 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0505 23:20:41.197185 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0505 23:20:41.197192 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0505 23:20:41.197199 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0505 23:20:41.197190 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0505 23:20:41.197523 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0505 23:20:41.197778 1 secure_serving.go:213] Serving securely on [::]:8443 I0505 23:20:41.197810 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0505 23:20:41.207420 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0505 23:20:41.207449 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0505 23:20:41.207547 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0505 23:20:41.213570 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0505 23:20:41.213589 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0505 23:20:41.219282 1 secretconfigobserver.go:119] support secret does not exist I0505 23:20:41.225258 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0505 23:20:41.230957 1 secretconfigobserver.go:119] support secret does not exist I0505 23:20:41.234527 1 recorder.go:161] Pruning old reports every 4h34m28s, max age is 288h0m0s I0505 23:20:41.240791 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0505 23:20:41.240802 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0505 23:20:41.240807 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0505 23:20:41.240811 1 insightsreport.go:296] Starting report retriever I0505 23:20:41.240819 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0505 23:20:41.240839 1 periodic.go:209] Running clusterconfig gatherer I0505 23:20:41.240883 1 tasks_processing.go:45] number of workers: 64 I0505 23:20:41.240911 1 tasks_processing.go:69] worker 2 listening for tasks. I0505 23:20:41.240917 1 tasks_processing.go:69] worker 14 listening for tasks. I0505 23:20:41.240923 1 tasks_processing.go:69] worker 3 listening for tasks. I0505 23:20:41.240927 1 tasks_processing.go:69] worker 0 listening for tasks. I0505 23:20:41.240928 1 tasks_processing.go:71] worker 14 working on container_runtime_configs task. I0505 23:20:41.240928 1 tasks_processing.go:69] worker 1 listening for tasks. I0505 23:20:41.240933 1 tasks_processing.go:71] worker 0 working on tsdb_status task. I0505 23:20:41.240957 1 tasks_processing.go:71] worker 1 working on machine_sets task. I0505 23:20:41.240965 1 tasks_processing.go:69] worker 41 listening for tasks. I0505 23:20:41.240964 1 tasks_processing.go:69] worker 39 listening for tasks. I0505 23:20:41.240972 1 tasks_processing.go:69] worker 42 listening for tasks. I0505 23:20:41.240951 1 tasks_processing.go:69] worker 52 listening for tasks. I0505 23:20:41.240980 1 tasks_processing.go:69] worker 44 listening for tasks. I0505 23:20:41.240975 1 tasks_processing.go:69] worker 43 listening for tasks. I0505 23:20:41.240982 1 tasks_processing.go:69] worker 58 listening for tasks. I0505 23:20:41.240992 1 tasks_processing.go:69] worker 16 listening for tasks. I0505 23:20:41.240994 1 tasks_processing.go:69] worker 46 listening for tasks. I0505 23:20:41.240987 1 tasks_processing.go:69] worker 15 listening for tasks. I0505 23:20:41.240990 1 tasks_processing.go:69] worker 45 listening for tasks. I0505 23:20:41.241000 1 tasks_processing.go:69] worker 53 listening for tasks. I0505 23:20:41.240999 1 tasks_processing.go:69] worker 17 listening for tasks. I0505 23:20:41.241005 1 tasks_processing.go:69] worker 18 listening for tasks. I0505 23:20:41.241010 1 tasks_processing.go:69] worker 48 listening for tasks. I0505 23:20:41.241005 1 tasks_processing.go:69] worker 47 listening for tasks. I0505 23:20:41.241014 1 tasks_processing.go:69] worker 59 listening for tasks. I0505 23:20:41.241019 1 tasks_processing.go:69] worker 4 listening for tasks. I0505 23:20:41.241024 1 tasks_processing.go:69] worker 54 listening for tasks. I0505 23:20:41.241022 1 tasks_processing.go:71] worker 2 working on operators_pods_and_events task. I0505 23:20:41.241027 1 tasks_processing.go:69] worker 51 listening for tasks. I0505 23:20:41.241030 1 tasks_processing.go:69] worker 60 listening for tasks. I0505 23:20:41.241034 1 tasks_processing.go:69] worker 32 listening for tasks. I0505 23:20:41.241037 1 tasks_processing.go:69] worker 62 listening for tasks. I0505 23:20:41.241038 1 tasks_processing.go:69] worker 56 listening for tasks. I0505 23:20:41.241042 1 tasks_processing.go:69] worker 26 listening for tasks. I0505 23:20:41.241039 1 tasks_processing.go:69] worker 6 listening for tasks. I0505 23:20:41.241039 1 tasks_processing.go:69] worker 57 listening for tasks. I0505 23:20:41.241052 1 tasks_processing.go:71] worker 43 working on schedulers task. I0505 23:20:41.240957 1 tasks_processing.go:69] worker 40 listening for tasks. I0505 23:20:41.241056 1 tasks_processing.go:71] worker 57 working on version task. I0505 23:20:41.241059 1 tasks_processing.go:71] worker 58 working on overlapping_namespace_uids task. I0505 23:20:41.241061 1 tasks_processing.go:69] worker 7 listening for tasks. I0505 23:20:41.241065 1 tasks_processing.go:71] worker 48 working on machines task. I0505 23:20:41.241067 1 tasks_processing.go:71] worker 4 working on oauths task. I0505 23:20:41.241071 1 tasks_processing.go:71] worker 7 working on image task. I0505 23:20:41.241073 1 tasks_processing.go:69] worker 38 listening for tasks. I0505 23:20:41.240998 1 tasks_processing.go:69] worker 61 listening for tasks. I0505 23:20:41.241070 1 tasks_processing.go:71] worker 45 working on networks task. I0505 23:20:41.241010 1 tasks_processing.go:69] worker 19 listening for tasks. I0505 23:20:41.241090 1 tasks_processing.go:69] worker 34 listening for tasks. I0505 23:20:41.241006 1 tasks_processing.go:69] worker 8 listening for tasks. I0505 23:20:41.241094 1 tasks_processing.go:69] worker 29 listening for tasks. I0505 23:20:41.241097 1 tasks_processing.go:71] worker 54 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0505 23:20:41.241104 1 tasks_processing.go:69] worker 36 listening for tasks. I0505 23:20:41.241106 1 tasks_processing.go:71] worker 42 working on image_pruners task. I0505 23:20:41.241106 1 tasks_processing.go:71] worker 46 working on clusterroles task. I0505 23:20:41.241114 1 tasks_processing.go:69] worker 37 listening for tasks. I0505 23:20:41.241121 1 tasks_processing.go:71] worker 47 working on ceph_cluster task. I0505 23:20:41.241110 1 tasks_processing.go:69] worker 11 listening for tasks. I0505 23:20:41.241135 1 tasks_processing.go:69] worker 10 listening for tasks. I0505 23:20:41.241132 1 tasks_processing.go:69] worker 9 listening for tasks. I0505 23:20:41.241143 1 tasks_processing.go:71] worker 59 working on storage_classes task. I0505 23:20:41.241152 1 tasks_processing.go:71] worker 39 working on nodenetworkconfigurationpolicies task. I0505 23:20:41.241058 1 tasks_processing.go:71] worker 17 working on openstack_controlplanes task. I0505 23:20:41.241019 1 tasks_processing.go:69] worker 21 listening for tasks. I0505 23:20:41.241022 1 tasks_processing.go:69] worker 50 listening for tasks. I0505 23:20:41.241024 1 tasks_processing.go:69] worker 22 listening for tasks. I0505 23:20:41.241029 1 tasks_processing.go:69] worker 31 listening for tasks. I0505 23:20:41.241024 1 tasks_processing.go:69] worker 63 listening for tasks. I0505 23:20:41.241030 1 tasks_processing.go:69] worker 5 listening for tasks. I0505 23:20:41.241032 1 tasks_processing.go:69] worker 55 listening for tasks. I0505 23:20:41.241045 1 tasks_processing.go:71] worker 60 working on machine_configs task. I0505 23:20:41.241061 1 tasks_processing.go:71] worker 18 working on aggregated_monitoring_cr_names task. I0505 23:20:41.241255 1 tasks_processing.go:71] worker 51 working on proxies task. I0505 23:20:41.241016 1 tasks_processing.go:69] worker 49 listening for tasks. I0505 23:20:41.241053 1 tasks_processing.go:71] worker 53 working on mutating_webhook_configurations task. I0505 23:20:41.241068 1 tasks_processing.go:69] worker 23 listening for tasks. I0505 23:20:41.241062 1 tasks_processing.go:71] worker 40 working on jaegers task. I0505 23:20:41.241303 1 tasks_processing.go:69] worker 13 listening for tasks. I0505 23:20:41.241315 1 tasks_processing.go:69] worker 12 listening for tasks. I0505 23:20:41.241321 1 tasks_processing.go:71] worker 15 working on sap_pods task. I0505 23:20:41.241050 1 tasks_processing.go:71] worker 32 working on openshift_machine_api_events task. I0505 23:20:41.241399 1 tasks_processing.go:71] worker 52 working on storage_cluster task. I0505 23:20:41.241405 1 tasks_processing.go:71] worker 44 working on cost_management_metrics_configs task. I0505 23:20:41.240928 1 tasks_processing.go:71] worker 3 working on sap_datahubs task. I0505 23:20:41.241571 1 tasks_processing.go:71] worker 10 working on cluster_apiserver task. I0505 23:20:41.241573 1 tasks_processing.go:71] worker 13 working on service_accounts task. I0505 23:20:41.241574 1 tasks_processing.go:71] worker 6 working on nodes task. I0505 23:20:41.241607 1 tasks_processing.go:71] worker 21 working on active_alerts task. I0505 23:20:41.241014 1 tasks_processing.go:69] worker 20 listening for tasks. I0505 23:20:41.241652 1 tasks_processing.go:71] worker 9 working on openstack_dataplanenodesets task. I0505 23:20:41.241655 1 tasks_processing.go:71] worker 20 working on lokistack task. I0505 23:20:41.241083 1 tasks_processing.go:69] worker 33 listening for tasks. I0505 23:20:41.241086 1 tasks_processing.go:69] worker 28 listening for tasks. I0505 23:20:41.241083 1 tasks_processing.go:69] worker 25 listening for tasks. I0505 23:20:41.241098 1 tasks_processing.go:69] worker 35 listening for tasks. I0505 23:20:41.241523 1 tasks_processing.go:71] worker 12 working on olm_operators task. I0505 23:20:41.241054 1 tasks_processing.go:71] worker 16 working on sap_config task. I0505 23:20:41.241075 1 tasks_processing.go:69] worker 24 listening for tasks. I0505 23:20:41.241536 1 tasks_processing.go:71] worker 19 working on validating_webhook_configurations task. I0505 23:20:41.241816 1 tasks_processing.go:71] worker 28 working on crds task. I0505 23:20:41.241821 1 tasks_processing.go:71] worker 25 working on feature_gates task. I0505 23:20:41.241830 1 tasks_processing.go:71] worker 33 working on operators task. I0505 23:20:41.241833 1 tasks_processing.go:71] worker 35 working on machine_healthchecks task. I0505 23:20:41.241528 1 tasks_processing.go:71] worker 38 working on pdbs task. I0505 23:20:41.241816 1 tasks_processing.go:71] worker 24 working on node_logs task. I0505 23:20:41.241539 1 tasks_processing.go:71] worker 34 working on metrics task. I0505 23:20:41.241547 1 tasks_processing.go:71] worker 29 working on silenced_alerts task. I0505 23:20:41.241551 1 tasks_processing.go:71] worker 31 working on container_images task. I0505 23:20:41.241553 1 tasks_processing.go:71] worker 22 working on nodenetworkstates task. I0505 23:20:41.241554 1 tasks_processing.go:71] worker 63 working on certificate_signing_requests task. I0505 23:20:41.241556 1 tasks_processing.go:71] worker 62 working on machine_config_pools task. I0505 23:20:41.241558 1 tasks_processing.go:71] worker 5 working on ingress_certificates task. I0505 23:20:41.241558 1 tasks_processing.go:71] worker 36 working on machine_autoscalers task. I0505 23:20:41.241560 1 tasks_processing.go:71] worker 56 working on openshift_logging task. I0505 23:20:41.241561 1 tasks_processing.go:71] worker 55 working on infrastructures task. I0505 23:20:41.241563 1 tasks_processing.go:71] worker 37 working on monitoring_persistent_volumes task. I0505 23:20:41.241565 1 tasks_processing.go:71] worker 49 working on qemu_kubevirt_launcher_logs task. I0505 23:20:41.241567 1 tasks_processing.go:71] worker 11 working on openstack_dataplanedeployments task. I0505 23:20:41.241568 1 tasks_processing.go:71] worker 23 working on image_registries task. I0505 23:20:41.241564 1 tasks_processing.go:71] worker 26 working on openstack_version task. I0505 23:20:41.241641 1 tasks_processing.go:71] worker 50 working on dvo_metrics task. I0505 23:20:41.241049 1 tasks_processing.go:71] worker 41 working on support_secret task. I0505 23:20:41.241077 1 tasks_processing.go:69] worker 27 listening for tasks. I0505 23:20:41.243741 1 tasks_processing.go:71] worker 27 working on pod_network_connectivity_checks task. I0505 23:20:41.241532 1 tasks_processing.go:71] worker 61 working on ingress task. I0505 23:20:41.241063 1 tasks_processing.go:69] worker 30 listening for tasks. I0505 23:20:41.243837 1 tasks_processing.go:71] worker 30 working on authentication task. I0505 23:20:41.241541 1 tasks_processing.go:71] worker 8 working on config_maps task. E0505 23:20:41.248751 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.236.135:9091: connect: connection refused I0505 23:20:41.248775 1 tasks_processing.go:71] worker 34 working on install_plans task. E0505 23:20:41.248794 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.236.135:9091: connect: connection refused E0505 23:20:41.248791 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.236.135:9091: connect: connection refused I0505 23:20:41.248826 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 6.738825ms to process 0 records E0505 23:20:41.248848 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.236.135:9091: connect: connection refused I0505 23:20:41.248865 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 7.850162ms to process 0 records I0505 23:20:41.248874 1 tasks_processing.go:74] worker 0 stopped. I0505 23:20:41.250399 1 tasks_processing.go:74] worker 40 stopped. I0505 23:20:41.250413 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 9.084932ms to process 0 records I0505 23:20:41.250635 1 tasks_processing.go:74] worker 1 stopped. I0505 23:20:41.250646 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 9.669909ms to process 0 records I0505 23:20:41.250652 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 9.226293ms to process 0 records I0505 23:20:41.250658 1 tasks_processing.go:74] worker 52 stopped. I0505 23:20:41.250680 1 tasks_processing.go:74] worker 44 stopped. I0505 23:20:41.250688 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 9.267015ms to process 0 records I0505 23:20:41.250795 1 tasks_processing.go:74] worker 48 stopped. E0505 23:20:41.250805 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0505 23:20:41.250813 1 gather.go:177] gatherer "clusterconfig" function "machines" took 9.71728ms to process 0 records I0505 23:20:41.250888 1 tasks_processing.go:74] worker 39 stopped. I0505 23:20:41.250899 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 9.730554ms to process 0 records I0505 23:20:41.250908 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 9.768162ms to process 0 records I0505 23:20:41.250914 1 tasks_processing.go:74] worker 47 stopped. I0505 23:20:41.251086 1 tasks_processing.go:74] worker 43 stopped. I0505 23:20:41.251185 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=a6b3b39265fe345851148bf2674453ec3cbebb2c8c1e3a674e3fb3f2dfe278b7 I0505 23:20:41.251199 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 10.02698ms to process 1 records I0505 23:20:41.251209 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 9.922969ms to process 0 records I0505 23:20:41.251220 1 tasks_processing.go:74] worker 17 stopped. I0505 23:20:41.251649 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0505 23:20:41.251664 1 controller.go:326] The initial operator extension status is healthy I0505 23:20:41.251672 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0505 23:20:41.251676 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0505 23:20:41.251679 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0505 23:20:41.251693 1 controller.go:489] The operator is still being initialized I0505 23:20:41.251701 1 controller.go:512] The operator is healthy I0505 23:20:41.251757 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0505 23:20:41.251772 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0505 23:20:41.251977 1 operator.go:288] started I0505 23:20:41.251998 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0505 23:20:41.264256 1 tasks_processing.go:74] worker 20 stopped. I0505 23:20:41.264275 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 22.586977ms to process 0 records I0505 23:20:41.289141 1 tasks_processing.go:74] worker 9 stopped. I0505 23:20:41.289168 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 47.470801ms to process 0 records I0505 23:20:41.289181 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 47.5917ms to process 0 records I0505 23:20:41.289188 1 tasks_processing.go:74] worker 3 stopped. I0505 23:20:41.289202 1 tasks_processing.go:74] worker 15 stopped. I0505 23:20:41.289222 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 47.868458ms to process 0 records I0505 23:20:41.289251 1 tasks_processing.go:74] worker 59 stopped. I0505 23:20:41.289350 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=cd11d853680c167dea661aad034a765833c415b1d56b89ad00c9e64c06dee593 I0505 23:20:41.289369 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=1a637a41eba8c7b5c303d3b97275ac82ca5698070f70937f390be0c2cf29a531 I0505 23:20:41.289375 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 48.097009ms to process 2 records I0505 23:20:41.289390 1 tasks_processing.go:74] worker 7 stopped. I0505 23:20:41.289471 1 recorder.go:75] Recording config/image with fingerprint=a196393ceb0e964525db72b0c0dfd39be4a09fbfa10a94e7354b4c6a179d392b I0505 23:20:41.289484 1 gather.go:177] gatherer "clusterconfig" function "image" took 48.312204ms to process 1 records I0505 23:20:41.289490 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 48.464612ms to process 0 records I0505 23:20:41.289494 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 47.266092ms to process 0 records I0505 23:20:41.289501 1 tasks_processing.go:74] worker 22 stopped. I0505 23:20:41.289505 1 tasks_processing.go:74] worker 14 stopped. I0505 23:20:41.289652 1 tasks_processing.go:74] worker 32 stopped. I0505 23:20:41.289661 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 48.297068ms to process 0 records I0505 23:20:41.289669 1 tasks_processing.go:74] worker 35 stopped. E0505 23:20:41.289675 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0505 23:20:41.289681 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 47.826786ms to process 0 records I0505 23:20:41.289708 1 tasks_processing.go:74] worker 10 stopped. I0505 23:20:41.289899 1 recorder.go:75] Recording config/apiserver with fingerprint=d7a78a0e409c6c2a423027a072f6dd7f26a2e3d61d511bd151efe3acc8daccc2 I0505 23:20:41.289914 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 48.1256ms to process 1 records I0505 23:20:41.289921 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 48.010459ms to process 0 records I0505 23:20:41.289928 1 tasks_processing.go:74] worker 16 stopped. I0505 23:20:41.289982 1 tasks_processing.go:74] worker 26 stopped. I0505 23:20:41.289998 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 46.916468ms to process 0 records I0505 23:20:41.290072 1 recorder.go:75] Recording config/proxy with fingerprint=92cc1373ded68e749dbab58599386bc987ebf8dd1ca8f22b258d9b17dea2287d I0505 23:20:41.290087 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 48.731052ms to process 1 records I0505 23:20:41.290107 1 tasks_processing.go:74] worker 51 stopped. I0505 23:20:41.290181 1 tasks_processing.go:74] worker 4 stopped. I0505 23:20:41.290358 1 recorder.go:75] Recording config/oauth with fingerprint=7ca60993df3b9037ab040ff8dc1545acd3b22a62ad8a07abf4bb1bf03c9bab20 I0505 23:20:41.290373 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 48.989603ms to process 1 records I0505 23:20:41.290450 1 tasks_processing.go:74] worker 42 stopped. I0505 23:20:41.290646 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=755bf4afcc4ec20a0a7c4d25518f1cc8342832094979fb9b807953bdc1a3ea45 I0505 23:20:41.290662 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 49.081112ms to process 1 records I0505 23:20:41.290670 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 47.530232ms to process 0 records I0505 23:20:41.290700 1 gather_logs.go:145] no pods in namespace were found I0505 23:20:41.290736 1 tasks_processing.go:74] worker 56 stopped. I0505 23:20:41.290777 1 recorder.go:75] Recording config/network with fingerprint=cbfaa8f6eb0506503cd87c860cbc6c4f5e3758ee6e52bfb7f62ef20da0c70c39 I0505 23:20:41.290789 1 gather.go:177] gatherer "clusterconfig" function "networks" took 49.186932ms to process 1 records I0505 23:20:41.290810 1 tasks_processing.go:74] worker 45 stopped. I0505 23:20:41.290881 1 tasks_processing.go:74] worker 55 stopped. I0505 23:20:41.291742 1 recorder.go:75] Recording config/infrastructure with fingerprint=c6b87dfb526506352c526d18b5dda44e7f394354a95c8195ad56312f5aec2819 I0505 23:20:41.291757 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 47.628965ms to process 1 records I0505 23:20:41.291764 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 47.876006ms to process 0 records E0505 23:20:41.291770 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0505 23:20:41.291777 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 47.111033ms to process 0 records I0505 23:20:41.291782 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 48.303718ms to process 0 records I0505 23:20:41.291785 1 tasks_processing.go:74] worker 49 stopped. I0505 23:20:41.291788 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 47.94421ms to process 0 records I0505 23:20:41.291795 1 tasks_processing.go:74] worker 11 stopped. I0505 23:20:41.291793 1 tasks_processing.go:74] worker 27 stopped. I0505 23:20:41.291805 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 48.92136ms to process 0 records I0505 23:20:41.291808 1 tasks_processing.go:74] worker 36 stopped. I0505 23:20:41.291820 1 tasks_processing.go:74] worker 24 stopped. I0505 23:20:41.291891 1 tasks_processing.go:74] worker 53 stopped. I0505 23:20:41.292381 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=fc2da9f70b9a1c3d7deab0d4dfafb3d5c55c91bafc1ef9dee756fd8e33a3a557 I0505 23:20:41.292452 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=6f2760af7d587d265df73f9eaa05395d9c7ca1aa97b7aa9caa0bd976640967d8 I0505 23:20:41.292695 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=5a40ff07d709dd18fdd63ebb927db046729887afc165bc690c041ae87ccf1782 I0505 23:20:41.292720 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 49.682265ms to process 3 records I0505 23:20:41.292736 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 48.331579ms to process 0 records I0505 23:20:41.292872 1 tasks_processing.go:74] worker 37 stopped. I0505 23:20:41.292966 1 tasks_processing.go:74] worker 30 stopped. I0505 23:20:41.293272 1 recorder.go:75] Recording config/authentication with fingerprint=0b80d2764f3f6ffaa5fd80df4e5c3f02eab378459e9c796977e6021409ce0c95 I0505 23:20:41.293331 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 47.656834ms to process 1 records I0505 23:20:41.293522 1 recorder.go:75] Recording config/featuregate with fingerprint=efd054f846c86d5239d402fb3620683888ff0178d29c1c2167c3733ff63312cd I0505 23:20:41.293552 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 49.838523ms to process 1 records I0505 23:20:41.293564 1 tasks_processing.go:74] worker 25 stopped. E0505 23:20:41.294114 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0505 23:20:41.294461 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 50.419215ms to process 0 records I0505 23:20:41.294058 1 tasks_processing.go:74] worker 41 stopped. I0505 23:20:41.294870 1 tasks_processing.go:74] worker 38 stopped. I0505 23:20:41.294986 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=84e556b8ca17db69e73351faf12d15527dfff6b197eaa0fa67c86b741fcf374d I0505 23:20:41.295015 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=ca6421ace0f7106ee8c2ebaab2286364a00e7d83b693839a180455ce1fd0a1f8 I0505 23:20:41.295038 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=6f7b7bb458747900639fe31d539b01f7eb9875fe3dabf7670562338b0ff261fd I0505 23:20:41.295063 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=249836353fe1db6cbba0dec51054be256de7ca323234c25885961f13b33b2abe I0505 23:20:41.295090 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=15c8b72df3aa6fbe021c1691f486496e4ec0814862cbe4fe18f3b0030fc99679 I0505 23:20:41.295117 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=0a4350cc9f9d269a4abacce50d1d5006557e71e8d9f9e1ec8433e8704e807a74 I0505 23:20:41.295117 1 controller.go:212] Source scaController *sca.Controller is not ready I0505 23:20:41.295129 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0505 23:20:41.295133 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0505 23:20:41.295137 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0505 23:20:41.295141 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0505 23:20:41.295143 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=aa981d34f80d1b5af6b84a798edbbddbe49993f346655e1e9ad25741fcfe8ca1 I0505 23:20:41.295164 1 controller.go:489] The operator is still being initialized I0505 23:20:41.295164 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=588177352a8eed5b585ad0f24fa9aae4aae00948dea8389f2e43c04e68e041af I0505 23:20:41.295173 1 controller.go:512] The operator is healthy I0505 23:20:41.295192 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=7a42d28a6547c7401e3bd23284627beadb038bf56e3ae54b15b9b25fa2eefdfb I0505 23:20:41.295220 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=2d3b3b663928f805794712eec9db21b93b870ec1727cf3d5c3fd06c5b0f8e369 I0505 23:20:41.295247 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=544eb02db3bafb9870cfc6b190e1b0b48dc8e7c024ae8bd441cbcdad1a13b6ea I0505 23:20:41.295259 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 52.895103ms to process 11 records I0505 23:20:41.295385 1 recorder.go:75] Recording config/ingress with fingerprint=5e2e15dfd1e1f7a1ed19cfe7e411937b2fc22d194a54d0c15813b2ccdfa772bb I0505 23:20:41.295396 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 51.08816ms to process 1 records I0505 23:20:41.295423 1 tasks_processing.go:74] worker 61 stopped. I0505 23:20:41.295491 1 tasks_processing.go:74] worker 6 stopped. I0505 23:20:41.295724 1 recorder.go:75] Recording config/node/ip-10-0-0-120.ec2.internal with fingerprint=1ed41a4393f273ca2aae67ca932083b9a388e0107dc6a4423ac881cfdf246ee2 I0505 23:20:41.295803 1 recorder.go:75] Recording config/node/ip-10-0-1-77.ec2.internal with fingerprint=8c912992300fdd89e40b0d394b59c6f4e668c2c6c3d286d256f1e7a14d6b9346 I0505 23:20:41.295880 1 recorder.go:75] Recording config/node/ip-10-0-2-33.ec2.internal with fingerprint=db311c0a3267122e1473517bb4cc0bb031834340950bdbdca4e57db5665caf41 I0505 23:20:41.295892 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 53.565548ms to process 3 records I0505 23:20:41.296395 1 tasks_processing.go:74] worker 23 stopped. I0505 23:20:41.296820 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=28748e1d994dfefb3381feee352664ffd944d89b767706fa5af0994a3fd15c15 I0505 23:20:41.296838 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 53.337573ms to process 1 records I0505 23:20:41.297226 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0505 23:20:41.297243 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0505 23:20:41.297293 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0505 23:20:41.298929 1 tasks_processing.go:74] worker 19 stopped. I0505 23:20:41.299365 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=3cdcf0f8d1fb77159561b64588d5906f2b29ba5e65fe36c1266f4479456542ee I0505 23:20:41.299480 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=023c47b4ffc4f908c16ba4835076df8f67d13c71d781d7278b46a1907d00cd71 I0505 23:20:41.299710 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=1173ac37873d733fabf95efa6e7b7eaafda6949ec7e331b41fd80e6fb00901e3 I0505 23:20:41.299764 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=7fc5f34d8a493d06b21c6b092a81e685c9802f0ef3f183a866adacb3b801e8c5 I0505 23:20:41.299805 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=ba41ded32a610ae060992939f8c87b8bdfc4ebb2410ea2f08acb8ab1b5f04bee I0505 23:20:41.299848 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=2165c56493de3a82803ec41f01b8069856b6a0cafa9a102f180ba6963d5c8e67 I0505 23:20:41.299896 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=87ac742fb5f2eea783fde94601b24e96fcd89fd687ce6373e14c23907876cad1 I0505 23:20:41.299961 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=6515eb67b0df6898c4101a451ecfceda4b355246e46266eca5fd675fd108149f I0505 23:20:41.300068 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=e7e655365a4bad4ff07d4ee7596753bb061ad24d352ff45093a3bf12f1147a12 I0505 23:20:41.300130 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=b18bb613dcad17b1fc6c46e1cd56569c04e6cc6d034ae2d15a120828f131980f I0505 23:20:41.300208 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=1a0855a994cdc15a59b94dd70eb928566b0381550b129a1f37ab625ec684869c I0505 23:20:41.300271 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=1738bfc9f92599c5e03424477a3da3c0e60cf17b588398e36440d5962d580884 I0505 23:20:41.300325 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=f30120bc14483bf80bef6e6511552bb267f25b365719cc658f623d4135b9d3db I0505 23:20:41.300337 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 57.181158ms to process 13 records I0505 23:20:41.300348 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 57.473669ms to process 0 records I0505 23:20:41.300370 1 tasks_processing.go:74] worker 63 stopped. I0505 23:20:41.301182 1 tasks_processing.go:74] worker 58 stopped. I0505 23:20:41.301222 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0505 23:20:41.301240 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 60.104072ms to process 1 records I0505 23:20:41.303438 1 tasks_processing.go:74] worker 50 stopped. I0505 23:20:41.303457 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=108780cb24f15bf572a4031966507398a0d0c6a28b3a008952f53b481dc191a9 I0505 23:20:41.303467 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 60.337052ms to process 1 records I0505 23:20:41.303568 1 tasks_processing.go:74] worker 46 stopped. I0505 23:20:41.303685 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=cf7da13c98548bd508a01db88e3c46390f9dc46f694dbe4db4d4a32049cbd532 I0505 23:20:41.303795 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=dd52830bc15af66659b9327e8a002d1fc63cc1ea6ca09548e9ce2570ebc3fbbf I0505 23:20:41.303805 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 62.335298ms to process 2 records I0505 23:20:41.308201 1 base_controller.go:82] Caches are synced for ConfigController I0505 23:20:41.308221 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0505 23:20:41.311874 1 tasks_processing.go:74] worker 12 stopped. I0505 23:20:41.311963 1 recorder.go:75] Recording config/olm_operators with fingerprint=b163eed755fd451e07cd10d7c49c23924b37ecc5f90552df60ee0eff25e5ea1a I0505 23:20:41.311978 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 70.14015ms to process 1 records I0505 23:20:41.312062 1 tasks_processing.go:74] worker 28 stopped. I0505 23:20:41.312485 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=b9fbb2790b01ff345a88d78d33b220a6e94c162e7349a38c6106af9a27942b4c I0505 23:20:41.312691 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=2a61b5474cd5e7765cccf460ed1d1a5799941c93ca7b12b136429255db58340b I0505 23:20:41.312701 1 gather.go:177] gatherer "clusterconfig" function "crds" took 70.049251ms to process 2 records I0505 23:20:41.312711 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 71.072176ms to process 0 records I0505 23:20:41.312717 1 tasks_processing.go:74] worker 18 stopped. I0505 23:20:41.315535 1 tasks_processing.go:74] worker 31 stopped. I0505 23:20:41.317099 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-7sjww with fingerprint=c2a8cc5befea735af904edbf78c80495a7a1551b3250a6944d9715a72c036645 I0505 23:20:41.317338 1 recorder.go:75] Recording config/running_containers with fingerprint=ae1a9cfd29d1a6f60608990caccf9b35134fdaca5af8770013721ea42dab1194 I0505 23:20:41.317404 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 73.368467ms to process 2 records I0505 23:20:41.320556 1 configmapobserver.go:84] configmaps "insights-config" not found I0505 23:20:41.324090 1 tasks_processing.go:74] worker 54 stopped. I0505 23:20:41.324106 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 82.97424ms to process 0 records I0505 23:20:41.336730 1 tasks_processing.go:74] worker 8 stopped. E0505 23:20:41.336750 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0505 23:20:41.336757 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0505 23:20:41.336761 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0505 23:20:41.336772 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0505 23:20:41.336802 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0505 23:20:41.336810 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0505 23:20:41.336814 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0505 23:20:41.336839 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0505 23:20:41.336880 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0505 23:20:41.336888 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0505 23:20:41.336894 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 92.86105ms to process 7 records E0505 23:20:41.336904 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0505 23:20:41.336909 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q42fl5um0ac787k2dpdmuh4vo7mjhnq-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q42fl5um0ac787k2dpdmuh4vo7mjhnq-primary-cert-bundle-secret" not found I0505 23:20:41.336922 1 tasks_processing.go:74] worker 5 stopped. I0505 23:20:41.336968 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=a5ebb115e8410f33b5d355662a18c091ece5f8cfd6e7d481f665fed722426659 I0505 23:20:41.336979 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 94.342056ms to process 1 records I0505 23:20:41.352163 1 base_controller.go:82] Caches are synced for LoggingSyncer I0505 23:20:41.352177 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0505 23:20:41.355791 1 tasks_processing.go:74] worker 57 stopped. I0505 23:20:41.356215 1 recorder.go:75] Recording config/version with fingerprint=27dd133c5834bb4fcb1bb90e07054d7b363f338385075f67676d4f1c33109592 I0505 23:20:41.356233 1 recorder.go:75] Recording config/id with fingerprint=9b468a5a4c2a7bbafb8450efeb7c19f9072c2489239f833aff8f3da9254d2a23 I0505 23:20:41.356240 1 gather.go:177] gatherer "clusterconfig" function "version" took 114.721982ms to process 2 records I0505 23:20:41.356252 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 113.6246ms to process 0 records I0505 23:20:41.356264 1 tasks_processing.go:74] worker 62 stopped. I0505 23:20:41.378166 1 tasks_processing.go:74] worker 60 stopped. I0505 23:20:41.378238 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0505 23:20:41.378254 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 136.912249ms to process 1 records I0505 23:20:41.386790 1 gather_cluster_operator_pods_and_events.go:121] Found 6 pods with 9 containers I0505 23:20:41.386804 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 2796202 bytes I0505 23:20:41.387024 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0505 23:20:41.387247 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-9mlvg pod in namespace openshift-dns (previous: false). I0505 23:20:41.388726 1 cluster_transfer.go:102] no available accepted cluster transfer I0505 23:20:41.388741 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0505 23:20:41.396775 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0505 23:20:41.407245 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0505 23:20:41.443888 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-9mlvg pod in namespace openshift-dns (previous: false). I0505 23:20:41.464900 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-bpb4w pod in namespace openshift-dns (previous: false). I0505 23:20:41.500315 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-bpb4w pod in namespace openshift-dns (previous: false). I0505 23:20:41.512472 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-t6tb8 pod in namespace openshift-dns (previous: false). I0505 23:20:41.544780 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-t6tb8 pod in namespace openshift-dns (previous: false). I0505 23:20:41.546289 1 sca.go:266] etc-pki-entitlement secret successfully created I0505 23:20:41.546304 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0505 23:20:41.560997 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-7tx6w pod in namespace openshift-dns (previous: false). I0505 23:20:41.573987 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-pnwvx pod in namespace openshift-dns (previous: false). I0505 23:20:41.718893 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-xkz9h pod in namespace openshift-dns (previous: false). I0505 23:20:41.917730 1 tasks_processing.go:74] worker 2 stopped. I0505 23:20:41.917823 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=1d702bcd9f2e9f1a4af02e21153cab863eddef1c134c4de17c2dfd26d6d9ad0f I0505 23:20:41.917913 1 recorder.go:75] Recording events/openshift-dns with fingerprint=2fc9f6d06902205732d739fa54351de347bae2fe84acc2a99a34ea5d7f8bc27b I0505 23:20:41.917923 1 recorder.go:75] Recording config/pod/openshift-dns/logs/dns-default-9mlvg/dns_current.log with fingerprint=0da3d1ba47c6bce118bfdc3775885285d51646387d15d8667b3c539d89e7e831 I0505 23:20:41.917931 1 recorder.go:75] Recording config/pod/openshift-dns/logs/dns-default-9mlvg/kube-rbac-proxy_current.log with fingerprint=f4618790a8aa9c4ebdbec560d6cdcdf2cccd009a27442ec2eca8a201a41abdf8 I0505 23:20:41.917958 1 recorder.go:75] Recording config/pod/openshift-dns/logs/dns-default-bpb4w/dns_current.log with fingerprint=e589eda5803ae5ccaa2f7a7d6cfd8fe55afe5aa8b63b3cdb77fe80d19bba5ace I0505 23:20:41.917965 1 recorder.go:75] Recording config/pod/openshift-dns/logs/dns-default-bpb4w/kube-rbac-proxy_current.log with fingerprint=e530d3e2d99d311ce9e914dc52bf121b766b823b769a02f62a2f53951b7ac0ac I0505 23:20:41.917970 1 recorder.go:75] Recording config/pod/openshift-dns/logs/dns-default-t6tb8/dns_current.log with fingerprint=644f8d3f66fbe57f2cbcd726bfd5a584f4516e262881225664a44524dabea7dc I0505 23:20:41.917976 1 recorder.go:75] Recording config/pod/openshift-dns/logs/dns-default-t6tb8/kube-rbac-proxy_current.log with fingerprint=528ddd26ad6420cb2e8f1cdce8c58f47371d6753d7954ae65dc2e91cde909ca3 I0505 23:20:41.917980 1 recorder.go:75] Recording config/pod/openshift-dns/logs/node-resolver-7tx6w/dns-node-resolver_current.log with fingerprint=f1492166c46a0fbc3e141e5f512d779c2300bcb64bc60612f83611f814cd9d87 I0505 23:20:41.917984 1 recorder.go:75] Recording config/pod/openshift-dns/logs/node-resolver-pnwvx/dns-node-resolver_current.log with fingerprint=c45112ae1c9bb3c8abd1a01ba99ae68c956890c4d65ff8e2b0a81cbaa50e7daf I0505 23:20:41.917988 1 recorder.go:75] Recording config/pod/openshift-dns/logs/node-resolver-xkz9h/dns-node-resolver_current.log with fingerprint=fb6af612f727d5ccef15ae5ac3c28af27b72817b41da791e07a2b9536ceeb3f4 I0505 23:20:41.917995 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 676.686125ms to process 11 records I0505 23:20:42.314542 1 tasks_processing.go:74] worker 29 stopped. I0505 23:20:42.314566 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0505 23:20:42.314578 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.072397985s to process 1 records I0505 23:20:42.314618 1 recorder.go:75] Recording config/alerts with fingerprint=111a1e73a8867f6fc0ad26f877ebb4a24797fc4efc0978f038a7a86689354a43 I0505 23:20:42.314624 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.072964088s to process 1 records I0505 23:20:42.314629 1 tasks_processing.go:74] worker 21 stopped. I0505 23:20:43.732363 1 tasks_processing.go:74] worker 33 stopped. I0505 23:20:43.732487 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=25449c8d0f65eedd92df970fee41aad0cdb626f1ddb608231d0176d93d11cce9 I0505 23:20:43.732538 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0505 23:20:43.732563 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=6fdde9de0c0340b0d2860ba1dddee09d3c9136eff8bb224f70622c9436103013 I0505 23:20:43.732573 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0505 23:20:43.732595 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=ba71fea77f90baf4b49eba3d0457572f17e2cf726c984bfd42298ed58962eb6c I0505 23:20:43.732612 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0505 23:20:43.732634 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=f515e30199d7e3649051733ce60bfd621f09f0a011a3d64d23f19fe4b887a9c7 I0505 23:20:43.732662 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=42e62af4bc10a22cf404afdcac331cfa004c834759acd5c33fa15859ce0ec009 I0505 23:20:43.732685 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=2ef98afc3a2c727e89eb20ec06a24e76c1c432d1272d67c09227c49f9b7d04b7 I0505 23:20:43.732716 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=55d8a8abe8cd7cc381e16f6be17201c666cbe20d4573a9f1ed78b698060010da I0505 23:20:43.733028 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=32fafb1f6f530f9da2f13185048906538e9fe68db029f7ddef7a39c006831280 I0505 23:20:43.733053 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=7685c264742bcb4d9dfe19d8ad5e4d3a3798ff0efd0201692d22b56589248f92 I0505 23:20:43.733065 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0505 23:20:43.733081 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=9c8ed4268abe5adadb0365b9ce26dec0e70da9fb80d4fa1d43c2ebc8cfb15c1f I0505 23:20:43.733092 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0505 23:20:43.733109 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=2bfbff8bd515a69305bbfa3e503c0df89378c7d51b61f299f9e3fe4c3d0e8abe I0505 23:20:43.733119 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0505 23:20:43.733138 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=326fa86366681bec847a5d9908e0290c948b4de9cb6152b723968175a09e11ab I0505 23:20:43.733149 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0505 23:20:43.733169 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=d15e24ef81b24454522cba095ae32eb2e8eaf768b98bc1b75f366a4d956c15f4 I0505 23:20:43.733271 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=c2999cf95cdb459d45830ead9744f467b3266c66ea9bd67f21dcf02088a911ea I0505 23:20:43.733280 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0505 23:20:43.733287 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0505 23:20:43.733308 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0505 23:20:43.733330 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=442908007be972529e5881d460f4b354f04a405e55b33eb4c2bc3411b0403c71 I0505 23:20:43.733356 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=20965de7a0d38df8fada8183f6a6073446d089b26062a024d224ec33ed96ea6e I0505 23:20:43.733367 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0505 23:20:43.733383 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=9a571942b04e3292d73c8364da135bc61356dcd40e24e0bb5df8f06f327af0d6 I0505 23:20:43.733393 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0505 23:20:43.733416 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=ae96a1a87269f4cc75c1a0f9759c13bf8a2713d844c8cfbadaacb306f7902210 I0505 23:20:43.733425 1 recorder.go:75] Recording config/clusteroperator/samples.operator.openshift.io/config/cluster with fingerprint=439a4284281b8dcef0621bb14ba23e2175a28a50613f86b33a171c49689474fd I0505 23:20:43.733437 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=b00850ebb4114a7245ecec88ff7543926aa10b49ff6383bc516488eb385452db I0505 23:20:43.733452 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=ba1ff65ebce26c85bb264947ed69743a4d7db6dbc2930d04d6451156eef61b5c I0505 23:20:43.733468 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=2a349c68e06685d580bb7a57425ceffed313742c1bc8fb7d839437ebab106ce8 I0505 23:20:43.733492 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=757c58463eb1a59b91958265a5ed5627bca562df3b88d53a694b68a2ab1a294f I0505 23:20:43.733501 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0505 23:20:43.733555 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=a94abd4c1c6460341d61528363e959308155a74003a2b2c04e60d4cf9a46079f I0505 23:20:43.733573 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0505 23:20:43.733582 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0505 23:20:43.733589 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.490513351s to process 39 records I0505 23:20:54.164159 1 tasks_processing.go:74] worker 34 stopped. I0505 23:20:54.164210 1 recorder.go:75] Recording config/installplans with fingerprint=95dfd5f33a9a46199239158a0ec0183b629818eca24ae41f4d706d7a3d604aec I0505 23:20:54.164222 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.915366426s to process 1 records I0505 23:20:54.852302 1 tasks_processing.go:74] worker 13 stopped. I0505 23:20:54.852674 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=f5738032f28aea743b389898fadee779c5cd56b79a04dcc121fb2c80f0e9fc94 I0505 23:20:54.852698 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.610707474s to process 1 records E0505 23:20:54.852785 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.611s with: function \"metrics\" failed with an error, function \"tsdb_status\" failed with an error, function \"machines\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error" I0505 23:20:54.853896 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "metrics" failed with an error, function "tsdb_status" failed with an error, function "machines" failed with an error, function "machine_healthchecks" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error I0505 23:20:54.853912 1 periodic.go:209] Running workloads gatherer I0505 23:20:54.853928 1 tasks_processing.go:45] number of workers: 2 I0505 23:20:54.853959 1 tasks_processing.go:69] worker 1 listening for tasks. I0505 23:20:54.853965 1 tasks_processing.go:71] worker 1 working on workload_info task. I0505 23:20:54.853974 1 tasks_processing.go:69] worker 0 listening for tasks. I0505 23:20:54.853989 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0505 23:20:54.863426 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-33.ec2.internal... I0505 23:20:54.863432 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-77.ec2.internal... I0505 23:20:54.863449 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-120.ec2.internal... I0505 23:20:54.900485 1 tasks_processing.go:74] worker 0 stopped. I0505 23:20:54.900510 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 46.476365ms to process 0 records I0505 23:20:59.241235 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 4s I0505 23:20:59.268447 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 24s for image data I0505 23:20:59.287415 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (20ms) I0505 23:20:59.303307 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (16ms) I0505 23:20:59.315026 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (12ms) I0505 23:20:59.326537 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (11ms) I0505 23:20:59.338280 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (12ms) I0505 23:20:59.349931 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (12ms) I0505 23:20:59.361692 1 gather_workloads_info.go:387] No image sha256:7bd3e88d6d3f878a89203df590b684c5a77bf08df4717451d3197ab74683a80e (12ms) I0505 23:20:59.373253 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (12ms) I0505 23:20:59.385408 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (12ms) I0505 23:20:59.397311 1 gather_workloads_info.go:387] No image sha256:f68c0da8268f23078f5f760a91a3c8212f2195acf18c231f6e034207a67adeec (12ms) I0505 23:20:59.409836 1 gather_workloads_info.go:387] No image sha256:0af73ad7a47c9ff53269ae2bc5163bf9e389085220c34e4bd42c92c4e7a85f6d (13ms) I0505 23:20:59.481001 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (71ms) I0505 23:20:59.581023 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (100ms) I0505 23:20:59.680744 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (100ms) I0505 23:20:59.783872 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (103ms) I0505 23:20:59.882933 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (99ms) I0505 23:20:59.986275 1 gather_workloads_info.go:387] No image sha256:1e65f9f5490fbc930d2dce96f03ddcb28990b839d7dc011ef5de5bec6c5095bf (103ms) I0505 23:21:00.081666 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (95ms) I0505 23:21:00.183418 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (102ms) I0505 23:21:00.289060 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (106ms) I0505 23:21:00.385029 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (96ms) I0505 23:21:00.484311 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (99ms) I0505 23:21:00.582248 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (98ms) I0505 23:21:00.682622 1 gather_workloads_info.go:387] No image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (100ms) I0505 23:21:00.782039 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (99ms) I0505 23:21:00.881962 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (100ms) I0505 23:21:00.989090 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (107ms) I0505 23:21:01.089216 1 gather_workloads_info.go:387] No image sha256:6e5df2aab9ce759e8f957341a11c087c233307e3c8f8760d21270efc19ccd3ce (100ms) I0505 23:21:01.182073 1 gather_workloads_info.go:387] No image sha256:101993f6e65d6fda8b90c583b718b720809182961f69bc777295f47e6df4d7ce (93ms) I0505 23:21:01.281271 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (99ms) I0505 23:21:01.382962 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (102ms) I0505 23:21:01.481669 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (99ms) I0505 23:21:01.581070 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (99ms) I0505 23:21:01.681523 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0505 23:21:01.781062 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0505 23:21:01.881100 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (100ms) I0505 23:21:01.979988 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (99ms) I0505 23:21:02.080728 1 gather_workloads_info.go:387] No image sha256:b99613494e9d6a4a8207323872d63cc02b830fc80a8bf972e62c64381aa4cd17 (101ms) I0505 23:21:02.180220 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (99ms) I0505 23:21:02.280786 1 gather_workloads_info.go:387] No image sha256:513b1166e376feca472fdd5174f6abf0abf02925d6a21b7b4d89a2ff2fc67bed (101ms) I0505 23:21:02.381589 1 gather_workloads_info.go:387] No image sha256:0092c6ff139f8698c91349d507eb3182bc5d6c4ae040f672e347ff504accdea2 (101ms) I0505 23:21:02.481622 1 gather_workloads_info.go:387] No image sha256:74713675d580a47a56b628469c4c05c41e114e0b935960d718ff0cf7e72f613d (100ms) I0505 23:21:02.584324 1 gather_workloads_info.go:387] No image sha256:90bdf9e678fe18e35401e34e1cb0912d822e9cb32b9e41cc545870540e9a87f2 (103ms) I0505 23:21:02.682357 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (98ms) I0505 23:21:02.783382 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (101ms) I0505 23:21:02.881392 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (98ms) I0505 23:21:02.981240 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0505 23:21:03.086888 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (106ms) I0505 23:21:03.186795 1 gather_workloads_info.go:387] No image sha256:1477f3287cc75352305274f248f781c1b7c2a8696f4f7bc450c45a4a5f5b10ea (100ms) I0505 23:21:03.186828 1 tasks_processing.go:74] worker 1 stopped. E0505 23:21:03.186839 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: received unexpected status code 500 Internal Server Error from https://10.129.0.14:8443/gather_runtime_info I0505 23:21:03.187131 1 recorder.go:75] Recording config/workload_info with fingerprint=d4ba233f58bb36d18ea470df89c267e692999737db645baea968bb2ca58b504c I0505 23:21:03.187145 1 gather.go:177] gatherer "workloads" function "workload_info" took 8.332855084s to process 1 records E0505 23:21:03.187169 1 periodic.go:247] "Unhandled Error" err="workloads failed after 8.333s with: function \"workload_info\" failed with an error" I0505 23:21:03.188271 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0505 23:21:03.188285 1 periodic.go:209] Running conditional gatherer I0505 23:21:03.197332 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0505 23:21:03.441749 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0505 23:21:04.474500 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0505 23:21:04.482870 1 conditional_gatherer.go:392] cluster version is '4.20.8' I0505 23:21:04.482906 1 tasks_processing.go:45] number of workers: 3 I0505 23:21:04.482918 1 tasks_processing.go:69] worker 2 listening for tasks. I0505 23:21:04.482922 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0505 23:21:04.482960 1 tasks_processing.go:69] worker 0 listening for tasks. I0505 23:21:04.482974 1 tasks_processing.go:69] worker 1 listening for tasks. I0505 23:21:04.482985 1 tasks_processing.go:74] worker 1 stopped. I0505 23:21:04.482978 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0505 23:21:04.482991 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0505 23:21:04.483022 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0505 23:21:04.483041 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 839ns to process 1 records I0505 23:21:04.483128 1 tasks_processing.go:74] worker 2 stopped. I0505 23:21:04.483136 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0505 23:21:04.483147 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.92µs to process 1 records I0505 23:21:04.483323 1 tasks_processing.go:74] worker 0 stopped. I0505 23:21:04.483336 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 326.845µs to process 0 records I0505 23:21:04.483364 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0505 23:21:04.483386 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0505 23:21:04.512030 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=2806386d307076276778cc0fbeba3784ce2f4656f9055a3aa23cb8f4a96101d4 I0505 23:21:04.512191 1 diskrecorder.go:70] Writing 122 records to /var/lib/insights-operator/insights-2026-05-05-232104.tar.gz I0505 23:21:04.519194 1 diskrecorder.go:51] Wrote 122 records to disk in 7ms I0505 23:21:04.519237 1 periodic.go:278] Gathering cluster info every 2h0m0s I0505 23:21:04.519265 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0505 23:22:11.241896 1 diskrecorder.go:223] Found files to send: insights-2026-05-05-232104.tar.gz I0505 23:22:11.241958 1 insightsuploader.go:150] Checking archives to upload periodically every 15m57.586052445s I0505 23:22:11.241967 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0505 23:22:11.256751 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0505 23:22:11.576703 1 requests.go:87] Successfully reported id=2026-05-05T23:22:11Z x-rh-insights-request-id=74b8b44048fb4859ab875a951eb959ba, wrote=52554 I0505 23:22:11.576781 1 insightsuploader.go:187] Uploaded report successfully in 334.801886ms I0505 23:22:11.576814 1 controller.go:128] Initializing last reported time to 2026-05-05T23:22:11Z I0505 23:22:11.576835 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0505 23:22:11.576842 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0505 23:22:11.576849 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0505 23:22:11.584252 1 controller.go:512] The operator is healthy I0505 23:22:41.260307 1 controller.go:512] The operator is healthy I0505 23:23:14.680124 1 insightsreport.go:137] Pulling report from smart-proxy I0505 23:23:14.680151 1 insightsreport.go:149] Retrieving report I0505 23:23:14.689096 1 requests.go:110] Retrieving report for cluster: 496ae723-3bd4-42a8-aa02-96fe73a119b5 I0505 23:23:14.689108 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/496ae723-3bd4-42a8-aa02-96fe73a119b5/reports I0505 23:23:14.692795 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/496ae723-3bd4-42a8-aa02-96fe73a119b5/reports I0505 23:23:14.913683 1 insightsreport.go:184] Report retrieved I0505 23:23:14.941638 1 insightsreport.go:239] Report retrieved correctly I0505 23:24:41.260539 1 controller.go:512] The operator is healthy I0505 23:25:41.231965 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0505 23:25:41.238992 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0505 23:25:41.239012 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0505 23:25:41.245004 1 secretconfigobserver.go:119] support secret does not exist I0505 23:26:41.291103 1 controller.go:512] The operator is healthy I0505 23:28:41.263396 1 controller.go:512] The operator is healthy I0505 23:30:41.227703 1 configmapobserver.go:84] configmaps "insights-config" not found I0505 23:30:41.245776 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0505 23:30:41.252388 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0505 23:30:41.252408 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0505 23:30:41.259418 1 controller.go:512] The operator is healthy I0505 23:30:41.259499 1 secretconfigobserver.go:119] support secret does not exist I0505 23:32:41.263744 1 controller.go:512] The operator is healthy