I0507 14:45:28.437577 1 cmd.go:253] Using service-serving-cert provided certificates I0507 14:45:28.438042 1 observer_polling.go:159] Starting file observer I0507 14:45:29.156226 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0507 14:45:29.156430 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0507 14:45:29.156990 1 secure_serving.go:57] Forcing use of http/1.1 only W0507 14:45:29.157010 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0507 14:45:29.157014 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0507 14:45:29.157017 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0507 14:45:29.157020 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0507 14:45:29.157023 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0507 14:45:29.157025 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0507 14:45:29.157022 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0507 14:45:29.160470 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0507 14:45:29.160489 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"6636e5f0-cf98-4dde-86f2-54760545a541", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0507 14:45:29.161862 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0507 14:45:29.161877 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0507 14:45:29.161878 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0507 14:45:29.161883 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0507 14:45:29.161892 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0507 14:45:29.161896 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0507 14:45:29.162116 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/var/run/secrets/serving-cert/tls.crt::/var/run/secrets/serving-cert/tls.key" I0507 14:45:29.162365 1 secure_serving.go:213] Serving securely on [::]:8443 I0507 14:45:29.162413 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0507 14:45:29.167259 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0507 14:45:29.167531 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0507 14:45:29.167554 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0507 14:45:29.175146 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0507 14:45:29.175167 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0507 14:45:29.182320 1 secretconfigobserver.go:119] support secret does not exist I0507 14:45:29.187420 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0507 14:45:29.191737 1 secretconfigobserver.go:119] support secret does not exist I0507 14:45:29.193544 1 recorder.go:161] Pruning old reports every 4h0m26s, max age is 288h0m0s I0507 14:45:29.198383 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0507 14:45:29.198412 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0507 14:45:29.198413 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0507 14:45:29.198421 1 insightsreport.go:296] Starting report retriever I0507 14:45:29.198426 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0507 14:45:29.198436 1 periodic.go:209] Running clusterconfig gatherer I0507 14:45:29.198492 1 tasks_processing.go:45] number of workers: 64 I0507 14:45:29.198529 1 tasks_processing.go:69] worker 2 listening for tasks. I0507 14:45:29.198525 1 tasks_processing.go:69] worker 6 listening for tasks. I0507 14:45:29.198537 1 tasks_processing.go:69] worker 5 listening for tasks. I0507 14:45:29.198541 1 tasks_processing.go:69] worker 4 listening for tasks. I0507 14:45:29.198546 1 tasks_processing.go:71] worker 5 working on pdbs task. I0507 14:45:29.198547 1 tasks_processing.go:69] worker 0 listening for tasks. I0507 14:45:29.198539 1 tasks_processing.go:71] worker 2 working on image_pruners task. I0507 14:45:29.198554 1 tasks_processing.go:71] worker 0 working on lokistack task. I0507 14:45:29.198557 1 tasks_processing.go:69] worker 9 listening for tasks. I0507 14:45:29.198555 1 tasks_processing.go:69] worker 3 listening for tasks. I0507 14:45:29.198564 1 tasks_processing.go:69] worker 10 listening for tasks. I0507 14:45:29.198548 1 tasks_processing.go:71] worker 4 working on schedulers task. I0507 14:45:29.198569 1 tasks_processing.go:69] worker 11 listening for tasks. I0507 14:45:29.198574 1 tasks_processing.go:69] worker 12 listening for tasks. I0507 14:45:29.198580 1 tasks_processing.go:69] worker 39 listening for tasks. I0507 14:45:29.198582 1 tasks_processing.go:69] worker 1 listening for tasks. I0507 14:45:29.198588 1 tasks_processing.go:69] worker 40 listening for tasks. I0507 14:45:29.198592 1 tasks_processing.go:69] worker 38 listening for tasks. I0507 14:45:29.198592 1 tasks_processing.go:69] worker 41 listening for tasks. I0507 14:45:29.198596 1 tasks_processing.go:69] worker 42 listening for tasks. I0507 14:45:29.198604 1 tasks_processing.go:69] worker 43 listening for tasks. I0507 14:45:29.198610 1 tasks_processing.go:69] worker 44 listening for tasks. I0507 14:45:29.198607 1 tasks_processing.go:69] worker 26 listening for tasks. I0507 14:45:29.198616 1 tasks_processing.go:69] worker 25 listening for tasks. I0507 14:45:29.198616 1 tasks_processing.go:69] worker 27 listening for tasks. I0507 14:45:29.198634 1 tasks_processing.go:69] worker 55 listening for tasks. I0507 14:45:29.198617 1 tasks_processing.go:69] worker 45 listening for tasks. I0507 14:45:29.198628 1 tasks_processing.go:69] worker 46 listening for tasks. I0507 14:45:29.198632 1 tasks_processing.go:69] worker 28 listening for tasks. I0507 14:45:29.198633 1 tasks_processing.go:69] worker 56 listening for tasks. I0507 14:45:29.198639 1 tasks_processing.go:69] worker 57 listening for tasks. I0507 14:45:29.198652 1 tasks_processing.go:69] worker 21 listening for tasks. I0507 14:45:29.198642 1 tasks_processing.go:69] worker 19 listening for tasks. I0507 14:45:29.198655 1 tasks_processing.go:69] worker 13 listening for tasks. I0507 14:45:29.198659 1 tasks_processing.go:69] worker 63 listening for tasks. I0507 14:45:29.198661 1 tasks_processing.go:69] worker 22 listening for tasks. I0507 14:45:29.198663 1 tasks_processing.go:69] worker 35 listening for tasks. I0507 14:45:29.198647 1 tasks_processing.go:69] worker 20 listening for tasks. I0507 14:45:29.198670 1 tasks_processing.go:69] worker 14 listening for tasks. I0507 14:45:29.198672 1 tasks_processing.go:69] worker 23 listening for tasks. I0507 14:45:29.198661 1 tasks_processing.go:69] worker 60 listening for tasks. I0507 14:45:29.198676 1 tasks_processing.go:69] worker 58 listening for tasks. I0507 14:45:29.198675 1 tasks_processing.go:69] worker 36 listening for tasks. I0507 14:45:29.198681 1 tasks_processing.go:69] worker 34 listening for tasks. I0507 14:45:29.198688 1 tasks_processing.go:69] worker 59 listening for tasks. I0507 14:45:29.198687 1 tasks_processing.go:69] worker 17 listening for tasks. I0507 14:45:29.198683 1 tasks_processing.go:69] worker 16 listening for tasks. I0507 14:45:29.198691 1 tasks_processing.go:69] worker 32 listening for tasks. I0507 14:45:29.198694 1 tasks_processing.go:69] worker 53 listening for tasks. I0507 14:45:29.198699 1 tasks_processing.go:69] worker 54 listening for tasks. I0507 14:45:29.198698 1 tasks_processing.go:69] worker 51 listening for tasks. I0507 14:45:29.198700 1 tasks_processing.go:69] worker 52 listening for tasks. I0507 14:45:29.198643 1 tasks_processing.go:69] worker 61 listening for tasks. I0507 14:45:29.198546 1 tasks_processing.go:69] worker 7 listening for tasks. I0507 14:45:29.198697 1 tasks_processing.go:69] worker 37 listening for tasks. I0507 14:45:29.198642 1 tasks_processing.go:69] worker 18 listening for tasks. I0507 14:45:29.198650 1 tasks_processing.go:69] worker 30 listening for tasks. I0507 14:45:29.198650 1 tasks_processing.go:69] worker 62 listening for tasks. I0507 14:45:29.198641 1 tasks_processing.go:69] worker 29 listening for tasks. I0507 14:45:29.198657 1 tasks_processing.go:69] worker 31 listening for tasks. I0507 14:45:29.198659 1 tasks_processing.go:69] worker 47 listening for tasks. I0507 14:45:29.198672 1 tasks_processing.go:69] worker 48 listening for tasks. I0507 14:45:29.198678 1 tasks_processing.go:69] worker 15 listening for tasks. I0507 14:45:29.198706 1 tasks_processing.go:69] worker 33 listening for tasks. I0507 14:45:29.198682 1 tasks_processing.go:69] worker 49 listening for tasks. I0507 14:45:29.198680 1 tasks_processing.go:69] worker 24 listening for tasks. I0507 14:45:29.198691 1 tasks_processing.go:69] worker 50 listening for tasks. I0507 14:45:29.198541 1 tasks_processing.go:71] worker 6 working on ingress_certificates task. I0507 14:45:29.198552 1 tasks_processing.go:69] worker 8 listening for tasks. I0507 14:45:29.198770 1 tasks_processing.go:71] worker 43 working on feature_gates task. I0507 14:45:29.198773 1 tasks_processing.go:71] worker 10 working on nodenetworkconfigurationpolicies task. I0507 14:45:29.198775 1 tasks_processing.go:71] worker 9 working on authentication task. I0507 14:45:29.198779 1 tasks_processing.go:71] worker 12 working on nodenetworkstates task. I0507 14:45:29.198778 1 tasks_processing.go:71] worker 3 working on overlapping_namespace_uids task. I0507 14:45:29.198781 1 tasks_processing.go:71] worker 39 working on sap_pods task. I0507 14:45:29.198790 1 tasks_processing.go:71] worker 36 working on openstack_version task. I0507 14:45:29.198794 1 tasks_processing.go:71] worker 57 working on operators task. I0507 14:45:29.198797 1 tasks_processing.go:71] worker 21 working on machine_configs task. I0507 14:45:29.198803 1 tasks_processing.go:71] worker 17 working on silenced_alerts task. I0507 14:45:29.198808 1 tasks_processing.go:71] worker 44 working on storage_classes task. I0507 14:45:29.198815 1 tasks_processing.go:71] worker 46 working on certificate_signing_requests task. I0507 14:45:29.198804 1 tasks_processing.go:71] worker 55 working on sap_config task. I0507 14:45:29.198905 1 tasks_processing.go:71] worker 54 working on image_registries task. I0507 14:45:29.198955 1 tasks_processing.go:71] worker 15 working on container_runtime_configs task. I0507 14:45:29.198795 1 tasks_processing.go:71] worker 34 working on pod_network_connectivity_checks task. I0507 14:45:29.198792 1 tasks_processing.go:71] worker 37 working on node_logs task. I0507 14:45:29.198799 1 tasks_processing.go:71] worker 59 working on image task. I0507 14:45:29.198904 1 tasks_processing.go:71] worker 13 working on monitoring_persistent_volumes task. I0507 14:45:29.198775 1 tasks_processing.go:71] worker 11 working on operators_pods_and_events task. I0507 14:45:29.198781 1 tasks_processing.go:71] worker 8 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0507 14:45:29.198786 1 tasks_processing.go:71] worker 58 working on olm_operators task. I0507 14:45:29.198788 1 tasks_processing.go:71] worker 60 working on active_alerts task. I0507 14:45:29.198888 1 tasks_processing.go:71] worker 25 working on clusterroles task. I0507 14:45:29.198874 1 tasks_processing.go:71] worker 45 working on proxies task. I0507 14:45:29.198895 1 tasks_processing.go:71] worker 63 working on machines task. I0507 14:45:29.198891 1 tasks_processing.go:71] worker 27 working on crds task. I0507 14:45:29.198891 1 tasks_processing.go:71] worker 26 working on networks task. I0507 14:45:29.198901 1 tasks_processing.go:71] worker 19 working on metrics task. I0507 14:45:29.198902 1 tasks_processing.go:71] worker 20 working on support_secret task. I0507 14:45:29.198906 1 tasks_processing.go:71] worker 35 working on machine_sets task. I0507 14:45:29.198911 1 tasks_processing.go:71] worker 22 working on qemu_kubevirt_launcher_logs task. I0507 14:45:29.198922 1 tasks_processing.go:71] worker 52 working on dvo_metrics task. I0507 14:45:29.198909 1 tasks_processing.go:71] worker 56 working on validating_webhook_configurations task. I0507 14:45:29.198916 1 tasks_processing.go:71] worker 14 working on openstack_dataplanedeployments task. I0507 14:45:29.198926 1 tasks_processing.go:71] worker 51 working on jaegers task. I0507 14:45:29.198928 1 tasks_processing.go:71] worker 28 working on ingress task. I0507 14:45:29.198928 1 tasks_processing.go:71] worker 61 working on version task. I0507 14:45:29.198938 1 tasks_processing.go:71] worker 32 working on cost_management_metrics_configs task. I0507 14:45:29.198939 1 tasks_processing.go:71] worker 41 working on sap_datahubs task. I0507 14:45:29.198936 1 tasks_processing.go:71] worker 23 working on openshift_logging task. I0507 14:45:29.198943 1 tasks_processing.go:71] worker 42 working on tsdb_status task. I0507 14:45:29.198770 1 tasks_processing.go:71] worker 1 working on ceph_cluster task. I0507 14:45:29.198941 1 tasks_processing.go:71] worker 16 working on service_accounts task. I0507 14:45:29.198942 1 tasks_processing.go:71] worker 40 working on machine_config_pools task. I0507 14:45:29.198948 1 tasks_processing.go:71] worker 47 working on infrastructures task. I0507 14:45:29.198945 1 tasks_processing.go:71] worker 38 working on mutating_webhook_configurations task. I0507 14:45:29.198951 1 tasks_processing.go:71] worker 33 working on oauths task. I0507 14:45:29.198946 1 tasks_processing.go:71] worker 7 working on install_plans task. I0507 14:45:29.198953 1 tasks_processing.go:71] worker 62 working on cluster_apiserver task. I0507 14:45:29.198957 1 tasks_processing.go:71] worker 18 working on storage_cluster task. I0507 14:45:29.198953 1 tasks_processing.go:71] worker 29 working on openstack_dataplanenodesets task. I0507 14:45:29.198955 1 tasks_processing.go:71] worker 48 working on openshift_machine_api_events task. I0507 14:45:29.198957 1 tasks_processing.go:71] worker 31 working on aggregated_monitoring_cr_names task. I0507 14:45:29.202930 1 tasks_processing.go:71] worker 12 working on config_maps task. I0507 14:45:29.202952 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 4.139035ms to process 0 records I0507 14:45:29.198960 1 tasks_processing.go:71] worker 30 working on openstack_controlplanes task. I0507 14:45:29.198964 1 tasks_processing.go:71] worker 24 working on machine_healthchecks task. I0507 14:45:29.198966 1 tasks_processing.go:71] worker 50 working on container_images task. I0507 14:45:29.198964 1 tasks_processing.go:71] worker 49 working on nodes task. I0507 14:45:29.198967 1 tasks_processing.go:71] worker 53 working on machine_autoscalers task. I0507 14:45:29.203256 1 tasks_processing.go:74] worker 0 stopped. I0507 14:45:29.203273 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 4.693472ms to process 0 records I0507 14:45:29.207137 1 tasks_processing.go:74] worker 34 stopped. E0507 14:45:29.207161 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0507 14:45:29.207174 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 8.075358ms to process 0 records E0507 14:45:29.207231 1 gather_prometheus_tsdb_status.go:49] Unable to tsdb status: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.96.64:9091: connect: connection refused E0507 14:45:29.207236 1 gather_most_recent_metrics.go:87] Unable to retrieve most recent metrics: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.96.64:9091: connect: connection refused I0507 14:45:29.207257 1 tasks_processing.go:74] worker 42 stopped. E0507 14:45:29.207266 1 gather.go:140] gatherer "clusterconfig" function "tsdb_status" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/status/tsdb": dial tcp 172.30.96.64:9091: connect: connection refused I0507 14:45:29.207279 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 5.239427ms to process 0 records E0507 14:45:29.207292 1 gather.go:140] gatherer "clusterconfig" function "metrics" failed with the error: Get "https://prometheus-k8s.openshift-monitoring.svc:9091/federate?match%5B%5D=cluster_installer&match%5B%5D=namespace%3Acontainer_cpu_usage%3Asum&match%5B%5D=namespace%3Acontainer_memory_usage_bytes%3Asum&match%5B%5D=vsphere_node_hw_version_total&match%5B%5D=virt_platform&match%5B%5D=console_helm_installs_total&match%5B%5D=console_helm_upgrades_total&match%5B%5D=console_helm_uninstalls_total&match%5B%5D=openshift_apps_deploymentconfigs_strategy_total&match%5B%5D=etcd_server_slow_apply_total&match%5B%5D=etcd_server_slow_read_indexes_total&match%5B%5D=haproxy_exporter_server_threshold": dial tcp 172.30.96.64:9091: connect: connection refused I0507 14:45:29.207314 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 6.377436ms to process 0 records I0507 14:45:29.207332 1 tasks_processing.go:74] worker 19 stopped. I0507 14:45:29.219693 1 tasks_processing.go:74] worker 55 stopped. I0507 14:45:29.219722 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 20.779719ms to process 0 records I0507 14:45:29.219829 1 tasks_processing.go:74] worker 63 stopped. E0507 14:45:29.219844 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0507 14:45:29.219858 1 gather.go:177] gatherer "clusterconfig" function "machines" took 19.714473ms to process 0 records I0507 14:45:29.220276 1 tasks_processing.go:74] worker 43 stopped. I0507 14:45:29.220441 1 recorder.go:75] Recording config/featuregate with fingerprint=940ff2b34744034dabd7df841f00a771c5cde904bbdd8c5ebfe1cc1a19db2a57 I0507 14:45:29.220459 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 21.493764ms to process 1 records I0507 14:45:29.220544 1 tasks_processing.go:74] worker 4 stopped. I0507 14:45:29.220550 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=d1af4890402d2eeccace8b5bc801fc4e23b736e79ff79757586457599bd29197 I0507 14:45:29.220564 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 21.870223ms to process 1 records I0507 14:45:29.220639 1 tasks_processing.go:74] worker 9 stopped. I0507 14:45:29.220755 1 recorder.go:75] Recording config/authentication with fingerprint=f3a15d8944c8b52d2a9ff3b2bb2b44d13638bbdf4360ba40f46eabfc0548805f I0507 14:45:29.220769 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 21.727944ms to process 1 records I0507 14:45:29.220785 1 tasks_processing.go:74] worker 15 stopped. I0507 14:45:29.220807 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 21.784945ms to process 0 records I0507 14:45:29.229407 1 tasks_processing.go:74] worker 59 stopped. I0507 14:45:29.229513 1 recorder.go:75] Recording config/image with fingerprint=944175dca6e061a375676f25de635cd794cc74189649aef069d29eff26e76153 I0507 14:45:29.229525 1 gather.go:177] gatherer "clusterconfig" function "image" took 29.892115ms to process 1 records I0507 14:45:29.229810 1 tasks_processing.go:74] worker 51 stopped. I0507 14:45:29.229836 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 28.131188ms to process 0 records I0507 14:45:29.229849 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0507 14:45:29.229877 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 28.093881ms to process 0 records I0507 14:45:29.229887 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 28.299031ms to process 0 records I0507 14:45:29.229892 1 controller.go:326] The initial operator extension status is healthy I0507 14:45:29.229897 1 tasks_processing.go:74] worker 32 stopped. I0507 14:45:29.229904 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0507 14:45:29.229912 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0507 14:45:29.229917 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0507 14:45:29.229940 1 controller.go:489] The operator is still being initialized I0507 14:45:29.229949 1 controller.go:512] The operator is healthy I0507 14:45:29.229894 1 tasks_processing.go:74] worker 14 stopped. I0507 14:45:29.230024 1 tasks_processing.go:74] worker 28 stopped. I0507 14:45:29.230021 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0507 14:45:29.230053 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0507 14:45:29.230211 1 operator.go:288] started I0507 14:45:29.230222 1 recorder.go:75] Recording config/ingress with fingerprint=0cffc574fb5e30134865f050eff510cf64f12df9d03ca098bf146bf082f49960 I0507 14:45:29.230237 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0507 14:45:29.230236 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 28.309842ms to process 1 records I0507 14:45:29.230258 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0507 14:45:29.230266 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 31.28402ms to process 1 records I0507 14:45:29.230272 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 28.251417ms to process 0 records I0507 14:45:29.230277 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 27.14506ms to process 0 records I0507 14:45:29.230285 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 26.906411ms to process 0 records I0507 14:45:29.230290 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 28.191751ms to process 0 records I0507 14:45:29.230292 1 tasks_processing.go:74] worker 41 stopped. I0507 14:45:29.230295 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 27.341162ms to process 0 records I0507 14:45:29.230302 1 tasks_processing.go:74] worker 3 stopped. I0507 14:45:29.230305 1 tasks_processing.go:74] worker 30 stopped. I0507 14:45:29.230300 1 tasks_processing.go:74] worker 29 stopped. I0507 14:45:29.230311 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 30.801514ms to process 0 records I0507 14:45:29.230300 1 tasks_processing.go:74] worker 53 stopped. I0507 14:45:29.230317 1 tasks_processing.go:74] worker 37 stopped. I0507 14:45:29.230305 1 tasks_processing.go:74] worker 23 stopped. E0507 14:45:29.230322 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0507 14:45:29.230328 1 tasks_processing.go:74] worker 24 stopped. I0507 14:45:29.230333 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 27.076101ms to process 0 records I0507 14:45:29.230339 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 31.476539ms to process 0 records I0507 14:45:29.230343 1 tasks_processing.go:74] worker 10 stopped. I0507 14:45:29.230618 1 tasks_processing.go:74] worker 35 stopped. I0507 14:45:29.230641 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 29.703668ms to process 0 records I0507 14:45:29.230739 1 tasks_processing.go:74] worker 5 stopped. I0507 14:45:29.230855 1 recorder.go:75] Recording config/pdbs/openshift-console/console with fingerprint=0eaebee523a259a3ea2f89b8e5ba734574b3ac8f8fa2696b90f072102038a6ae I0507 14:45:29.230893 1 recorder.go:75] Recording config/pdbs/openshift-console/downloads with fingerprint=203f4bb371821502537d85a813b45c3797047ba4e4571392d6ebd0c2e796e4cd I0507 14:45:29.230916 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=fbbfd61251f7543d243d7a4f2008e51f1fa3a4ebbbf501e448c6674a17914474 I0507 14:45:29.230944 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=c52d45ba83105be062c3462a1a0a123a20dec79a921bbf38368a860f8ca34eee I0507 14:45:29.230971 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/alertmanager-main with fingerprint=edf87979d8bbd3e9691bf8d2983ac20faca5993c946ad29fde35b19d37364622 I0507 14:45:29.230996 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/metrics-server with fingerprint=c5d3966a1605980c31373d7a6855da38e04ae1b343d9cd27076b910663f8001e I0507 14:45:29.231021 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/monitoring-plugin with fingerprint=e78fbd2386022a3e49a7060e64e8171f272054beb6213ee756c58a40bf08c6bf I0507 14:45:29.231047 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-k8s with fingerprint=a8f92ff4f611dc3857cd8971ae596e370ac99e13b1bc22b68f1da03ce03b948b I0507 14:45:29.231071 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/prometheus-operator-admission-webhook with fingerprint=da114667b2e089c651fc6a0703a211b072e285c93116a3197dd7d661b66af9fa I0507 14:45:29.231099 1 recorder.go:75] Recording config/pdbs/openshift-monitoring/thanos-querier-pdb with fingerprint=0b0c4f95ab97c4b82953cbbb251213886619e2a8d99fa11494162d6732add0cd I0507 14:45:29.231124 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=e0b14be2c59cd0b60eb4b1a61de6f2d40c56826a916d9d4db51dbf6a59c8f673 I0507 14:45:29.231136 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 32.18533ms to process 11 records I0507 14:45:29.231152 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 32.126266ms to process 0 records I0507 14:45:29.231158 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 32.17442ms to process 0 records I0507 14:45:29.231166 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 29.022663ms to process 0 records I0507 14:45:29.231175 1 tasks_processing.go:74] worker 1 stopped. I0507 14:45:29.231182 1 tasks_processing.go:74] worker 36 stopped. I0507 14:45:29.231188 1 tasks_processing.go:74] worker 39 stopped. I0507 14:45:29.232922 1 tasks_processing.go:74] worker 20 stopped. E0507 14:45:29.232933 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0507 14:45:29.232941 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 32.024483ms to process 0 records I0507 14:45:29.237416 1 tasks_processing.go:74] worker 18 stopped. I0507 14:45:29.237428 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 34.634338ms to process 0 records I0507 14:45:29.238298 1 tasks_processing.go:74] worker 26 stopped. I0507 14:45:29.239339 1 recorder.go:75] Recording config/network with fingerprint=001a323b9a9a457c8f7fa6d3ed58cc3e73629cdd9c88aca0490b85b835971181 I0507 14:45:29.239414 1 gather.go:177] gatherer "clusterconfig" function "networks" took 37.538589ms to process 1 records I0507 14:45:29.250413 1 tasks_processing.go:74] worker 25 stopped. I0507 14:45:29.250720 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=5387458b4d8bda1a1b2b55c504e258e7de2d2b85f4c15e23b66515e73bc0c8d9 I0507 14:45:29.250879 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=75c143d9a9ea5cb53ee264eadb9984c46f81ffbcc16ebd46296cbd082cc85e49 I0507 14:45:29.250894 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 50.438662ms to process 2 records I0507 14:45:29.250929 1 tasks_processing.go:74] worker 27 stopped. I0507 14:45:29.251040 1 gather_logs.go:145] no pods in namespace were found I0507 14:45:29.251333 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0507 14:45:29.251350 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0507 14:45:29.251355 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0507 14:45:29.251362 1 controller.go:212] Source scaController *sca.Controller is not ready I0507 14:45:29.251367 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0507 14:45:29.251417 1 controller.go:489] The operator is still being initialized I0507 14:45:29.251426 1 controller.go:512] The operator is healthy I0507 14:45:29.252274 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=7c4795f747a9b13ca94ef2d5497bdbdf107609c37a5663a62095fa033cca5710 I0507 14:45:29.252798 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=8cb7431f99009de012885b5098eeee44ca08b7743dbf3d649a0e0e4eb964e27e I0507 14:45:29.252861 1 gather.go:177] gatherer "clusterconfig" function "crds" took 50.607875ms to process 2 records I0507 14:45:29.253213 1 tasks_processing.go:74] worker 33 stopped. I0507 14:45:29.253996 1 recorder.go:75] Recording config/oauth with fingerprint=3d8e8f6819e5028c8e40de421a9ce2f4d6d60ac6cd608ce080f805fc83c5502b I0507 14:45:29.254063 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 48.305097ms to process 1 records I0507 14:45:29.255063 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 50.083568ms to process 0 records I0507 14:45:29.255100 1 tasks_processing.go:74] worker 22 stopped. I0507 14:45:29.255207 1 tasks_processing.go:74] worker 47 stopped. I0507 14:45:29.255858 1 recorder.go:75] Recording config/infrastructure with fingerprint=c6cfdd2a97467df65ab0a87bd671f2b2807579c89899513b0a64bc646294ecec I0507 14:45:29.255877 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 48.78531ms to process 1 records I0507 14:45:29.255956 1 tasks_processing.go:74] worker 62 stopped. I0507 14:45:29.256046 1 recorder.go:75] Recording config/apiserver with fingerprint=5db6151ad659200b8099e19efa55a0e0ca866b728a0d3a9efb91ece81fa32ed6 I0507 14:45:29.256058 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 48.7018ms to process 1 records I0507 14:45:29.256064 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 49.873396ms to process 0 records I0507 14:45:29.256075 1 tasks_processing.go:74] worker 48 stopped. I0507 14:45:29.256152 1 tasks_processing.go:74] worker 38 stopped. I0507 14:45:29.256195 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=7527dc59c8332d29d188f88be3d7bffbf5d62c768733d398a1a1c2f1ff1e5054 W0507 14:45:29.256212 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0507 14:45:29.256225 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=e32f0a4c30c94baca58b54e8750da724b4912d2df3cb94b10967c2a5388fd520 I0507 14:45:29.256248 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=75a4efb377d7f557b0afee9f2f3a6fdf3e86aa6c4715a90b34c0ab55a2adc658 I0507 14:45:29.256255 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 51.497151ms to process 3 records I0507 14:45:29.256319 1 recorder.go:75] Recording config/validatingwebhookconfigurations/alertmanagerconfigs.openshift.io with fingerprint=196415ccb7ca9168a9b147cf3bff79385b4d9f5fa6d81cff7fb8cf72acdfc7b4 I0507 14:45:29.256345 1 tasks_processing.go:74] worker 56 stopped. I0507 14:45:29.256369 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=f764b84df7a2c15583979031734cb31fdcccee482fd50a25793272306e663f29 I0507 14:45:29.256460 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=2d250a96c0734ee3d3e2ea73757c7182ea38a807f5ff6433492c5dd97a4ed16a I0507 14:45:29.256486 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=1ae42120b8afc556abc3b499b46c72cbfb08b5a5391a5b3dd82f9db42e6f338f I0507 14:45:29.256509 1 recorder.go:75] Recording config/validatingwebhookconfigurations/prometheusrules.openshift.io with fingerprint=bb928594fcb1a813de9f97cb3e721ce674bf5274f3df6d80ddc66f4204f06fd5 I0507 14:45:29.256533 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=67f8db911310b665211adee5725bec98812c7fb12dee01ef4fa467d986542d28 I0507 14:45:29.256562 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=17e15435aa7308a2d7dc5efc0639bee160137c43fe1686f11482eeb2971ba830 I0507 14:45:29.256587 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=f011feba96f216aabb9f078320bdfa467efb9ba904ceccd335cb938aff858dc4 I0507 14:45:29.256610 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=08f1edcc327bf663692da673d3cccd58a5e876d08352f9efa3842c14c15d2c89 I0507 14:45:29.256643 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=915f961c2b704de6e7d8cbcbe156cb576e6c0951be5c1040175bdc0868ea9176 I0507 14:45:29.256667 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=2b801fded6b3d76a8641cd95e76f6b2d5e0c1358c12476fb8bedf04ded7866c0 I0507 14:45:29.256693 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=436248eb3e5937f8d5e17eb5a2324f96a873e438cf503b622523cbaa0c1dc433 I0507 14:45:29.256719 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=fd6f90a8126bad4854e771865ddff6b46f5b9fdbca4910f5b95493f0e250c204 I0507 14:45:29.256727 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 53.764643ms to process 13 records I0507 14:45:29.256807 1 tasks_processing.go:74] worker 49 stopped. I0507 14:45:29.257002 1 recorder.go:75] Recording config/node/ip-10-0-0-220.ec2.internal with fingerprint=33eca2af21b71fc5a43acb91f857426ced86d17cd49eb0abed407ec467fdf2c0 I0507 14:45:29.257055 1 recorder.go:75] Recording config/node/ip-10-0-1-217.ec2.internal with fingerprint=d2ecc1379b7aea1ce6b235456cbf2f1d0de020fa05ef342ca36f0c277ff2edc2 I0507 14:45:29.257114 1 recorder.go:75] Recording config/node/ip-10-0-2-38.ec2.internal with fingerprint=2d40ae1d0e4b50e162b8b784007b997d9885af9e72d95d71db9194b333a9d932 I0507 14:45:29.257122 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 52.15186ms to process 3 records I0507 14:45:29.257137 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 56.385746ms to process 0 records I0507 14:45:29.257215 1 tasks_processing.go:74] worker 13 stopped. I0507 14:45:29.257274 1 tasks_processing.go:74] worker 2 stopped. I0507 14:45:29.257312 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=5c6c075514414f9a86918c2e6ba99fc83f8f8a0f676bccbf113483a14ff974aa I0507 14:45:29.257322 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 57.856147ms to process 1 records I0507 14:45:29.260781 1 tasks_processing.go:74] worker 40 stopped. I0507 14:45:29.260797 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 58.27981ms to process 0 records I0507 14:45:29.261917 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0507 14:45:29.261943 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0507 14:45:29.262298 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0507 14:45:29.265501 1 tasks_processing.go:74] worker 46 stopped. I0507 14:45:29.265521 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 66.661458ms to process 0 records I0507 14:45:29.265588 1 tasks_processing.go:74] worker 58 stopped. I0507 14:45:29.265663 1 recorder.go:75] Recording config/olm_operators with fingerprint=22e54d725916b723243452e5e6b5a1f4820580f8d253bef6f5fe6e3857577d6b I0507 14:45:29.265682 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 65.880026ms to process 1 records I0507 14:45:29.266158 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload I0507 14:45:29.266457 1 tasks_processing.go:74] worker 45 stopped. I0507 14:45:29.266518 1 recorder.go:75] Recording config/proxy with fingerprint=24f91517abd40d69043504a0a6cb7716a388aad394b57b89379dd29ad265747b I0507 14:45:29.266530 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 66.450153ms to process 1 records I0507 14:45:29.267320 1 tasks_processing.go:74] worker 44 stopped. I0507 14:45:29.267430 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=ae94f90b91eca66ff45168ae30143b7467f15b6b8fe4c230ddc0fbfa2c62d1b7 I0507 14:45:29.267448 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=84ddf5e762b367fec864d6817f45fc9761832963f32d51458dca1fa743706b89 I0507 14:45:29.267454 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 68.496453ms to process 2 records I0507 14:45:29.267496 1 tasks_processing.go:74] worker 54 stopped. I0507 14:45:29.267826 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=96473d48900e12d0f0e74d6f1e5aa2e44beabc872e99561d88e037744bc3f4b9 I0507 14:45:29.267838 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 68.52898ms to process 1 records I0507 14:45:29.267861 1 base_controller.go:82] Caches are synced for ConfigController I0507 14:45:29.267868 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0507 14:45:29.268331 1 tasks_processing.go:74] worker 61 stopped. I0507 14:45:29.268602 1 recorder.go:75] Recording config/version with fingerprint=4712709bf1e8cffdf8c3d7105f9fdb57fb7293c8c290f4bc32dd84b3d3168411 I0507 14:45:29.268618 1 recorder.go:75] Recording config/id with fingerprint=9ae620bcc42f9698e489553222cbf72155e47ea5057c3ad0f25c04f67c50463c I0507 14:45:29.268625 1 gather.go:177] gatherer "clusterconfig" function "version" took 66.574889ms to process 2 records I0507 14:45:29.272482 1 configmapobserver.go:84] configmaps "insights-config" not found I0507 14:45:29.273156 1 tasks_processing.go:74] worker 31 stopped. I0507 14:45:29.273175 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 70.270346ms to process 0 records I0507 14:45:29.274170 1 tasks_processing.go:74] worker 6 stopped. E0507 14:45:29.274184 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0507 14:45:29.274191 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q552lgi3q7l1ua5jmhrf58d1ggu8u50-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q552lgi3q7l1ua5jmhrf58d1ggu8u50-primary-cert-bundle-secret" not found I0507 14:45:29.274241 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=ba0da3b2eefdc79ff005460672469c42b97237525a11ba679fbacfb0293340a2 I0507 14:45:29.274252 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 75.39775ms to process 1 records I0507 14:45:29.278615 1 tasks_processing.go:74] worker 8 stopped. I0507 14:45:29.278640 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 78.980303ms to process 0 records I0507 14:45:29.286453 1 tasks_processing.go:74] worker 12 stopped. E0507 14:45:29.286472 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0507 14:45:29.286478 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0507 14:45:29.286484 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0507 14:45:29.286498 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0507 14:45:29.286528 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0507 14:45:29.286533 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0507 14:45:29.286538 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0507 14:45:29.286561 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=34cdf5d36d45a9e3ee7c2be31546e3d937ae4ebf144ac07b0210c06a1610a94d I0507 14:45:29.286639 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0507 14:45:29.286652 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0507 14:45:29.286659 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 83.499461ms to process 7 records I0507 14:45:29.289422 1 tasks_processing.go:74] worker 50 stopped. I0507 14:45:29.290690 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-twnk6 with fingerprint=9722d88e02f17dfdf2c67b0816ce8e77157422af099de5e27f722e931c8794a2 I0507 14:45:29.290772 1 recorder.go:75] Recording config/running_containers with fingerprint=f78cfb37a2f578dc78c7d79702b90c92e6b7a1514e3f9c225712c67d0b3a61aa I0507 14:45:29.290783 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 86.284786ms to process 2 records I0507 14:45:29.330750 1 base_controller.go:82] Caches are synced for LoggingSyncer I0507 14:45:29.330770 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0507 14:45:29.342915 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0507 14:45:29.347652 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0507 14:45:29.352354 1 tasks_processing.go:74] worker 21 stopped. I0507 14:45:29.352408 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0507 14:45:29.352420 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 153.539584ms to process 1 records I0507 14:45:29.427179 1 cluster_transfer.go:102] no available accepted cluster transfer I0507 14:45:29.427200 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=NoClusterTransfer message=no available cluster transfer I0507 14:45:29.525214 1 sca.go:266] etc-pki-entitlement secret successfully created I0507 14:45:29.525236 1 controllerstatus.go:80] name=scaController healthy=true reason=Updated message=SCA certs successfully updated I0507 14:45:30.074762 1 gather_cluster_operator_pods_and_events.go:121] Found 39 pods with 87 containers I0507 14:45:30.074782 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 289262 bytes I0507 14:45:30.074855 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for init-config-reloader container alertmanager-main-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:30.244925 1 tasks_processing.go:74] worker 17 stopped. I0507 14:45:30.244953 1 recorder.go:75] Recording config/silenced_alerts.json with fingerprint=37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570 I0507 14:45:30.244969 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 1.046101174s to process 1 records I0507 14:45:30.245878 1 tasks_processing.go:74] worker 60 stopped. I0507 14:45:30.245917 1 recorder.go:75] Recording config/alerts with fingerprint=92d19018ed1124b9a3e158cba59b968a5b5f1bae13e932e791abf775c2748746 I0507 14:45:30.245930 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.045999319s to process 1 records W0507 14:45:30.256357 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0507 14:45:30.280986 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for alertmanager container alertmanager-main-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:30.489517 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for config-reloader container alertmanager-main-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:30.679297 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-web container alertmanager-main-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:30.879033 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container alertmanager-main-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:31.079489 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-metric container alertmanager-main-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:31.090550 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found W0507 14:45:31.255866 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0507 14:45:31.278266 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for prom-label-proxy container alertmanager-main-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:31.479027 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for init-config-reloader container alertmanager-main-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:31.679714 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for alertmanager container alertmanager-main-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:31.693101 1 tasks_processing.go:74] worker 57 stopped. I0507 14:45:31.693251 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=8aee483bc205abf5594d095ebba5302944dcfb001b47926a3839ce20e9c705b3 I0507 14:45:31.693298 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/console/cluster with fingerprint=be84fc75c87766f0f4d4e9fb1f9db618d008a21cc463e8239af81a1c296612d1 I0507 14:45:31.693322 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=edfb82aa289808b536a6481878c5a62bfa5be7f418433f2520fdc1b38082cf6a I0507 14:45:31.693332 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0507 14:45:31.693357 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=2b9197bf5b75164dbaab8cf2dcd98529fbb687ce563ef99e0ec766f917d88c06 I0507 14:45:31.693374 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0507 14:45:31.693418 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=fc9e2db7a5c5d89e3a3d40cd8349a9f7c6327227e903f1298cff2b960c668e6c I0507 14:45:31.693443 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=f5eb0850a110646c6ce9bbc02069eafa6a3edead3248acb7b070c6260772b729 I0507 14:45:31.693468 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=e7a558fecdf826e5a6d50cb13beae64554b9f79446bf7660d503a2f5b35b6f39 I0507 14:45:31.693500 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=75e88220801e3f3da910939fe8e8f691596d7a17785517402d59534c91447025 I0507 14:45:31.693785 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/insightsoperator/cluster with fingerprint=5ff81b08bf4a9422e3d2de3682927e2e3931ada2a7bc6751637647bbe4cbc55f I0507 14:45:31.693807 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=747c9b595dd562956ba399ef6a659d5e762832d363b6f116f87f5fc2fefbbcce I0507 14:45:31.693816 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0507 14:45:31.693831 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=c65e853231266dd80e3c28175579cd5f053fb528b628a39eb9dd9aa2090a0578 I0507 14:45:31.693841 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0507 14:45:31.693856 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=038bdf5ab8bc5131a11d44a25494372650bf9353a3dfd939b90fa8e90b5da65a I0507 14:45:31.693865 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0507 14:45:31.693885 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=a458088c1602c3cf0e9280fb0d36bdac41fc23acda73defd5299080e0f89ccc8 I0507 14:45:31.693895 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0507 14:45:31.693915 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=d38782b54d78df20b1592aef4d0542be64e899e5b7cc57d6c9b29634b13600b9 I0507 14:45:31.694037 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=4b6d773e201dd0c624151d8b35902e0d49dafd933d2dcd6757c95ded55fb3ea5 I0507 14:45:31.694047 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0507 14:45:31.694055 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0507 14:45:31.694078 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0507 14:45:31.694100 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=7264c8dfeb177186605db4837e9b69104eb0cd8e665e4cf11a11de322088f09f I0507 14:45:31.694124 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=803cd2f5b7e06af0f346b146bf25fb539606af868f0c37a88205637480d03c33 I0507 14:45:31.694134 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0507 14:45:31.694149 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=505fad911242dbf83b9782cdae24a619b631e58169af35f390fc9dbc2c4214fe I0507 14:45:31.694160 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0507 14:45:31.694172 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=56391ff9888f694bb2d379da809ab667802f4b6ee49348a951cd744553a2eb4d I0507 14:45:31.694186 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=0e425dc32a3ed2a49a12574576e0fa1a3006ef89d7f4c8fa8315cf797b9ed721 I0507 14:45:31.694200 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=174e89611805dd333af2c5a2830dd0a09516b24be24da3a1889b3cbe9d8c1594 I0507 14:45:31.694216 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=d876d906f79a720221c304771c53de04dc854c81433dad55d001764d21d94466 I0507 14:45:31.694237 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=56bc2c13aced00cc8c3319d4b3fd944de1a40a8d799fafaa51d16df288470a27 I0507 14:45:31.694246 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0507 14:45:31.694270 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=5e9fd08c7e79c25696d145fa5db4a4a0e2ab1986efdb7c924dc063805e7224e7 I0507 14:45:31.694288 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0507 14:45:31.694297 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0507 14:45:31.694303 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.494289212s to process 38 records I0507 14:45:31.879298 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for config-reloader container alertmanager-main-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:32.078267 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-web container alertmanager-main-1 pod in namespace openshift-monitoring (previous: false). W0507 14:45:32.255722 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0507 14:45:32.278781 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container alertmanager-main-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:32.478546 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-metric container alertmanager-main-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:32.678964 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for prom-label-proxy container alertmanager-main-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:32.880859 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for cluster-monitoring-operator container cluster-monitoring-operator-698978dbbf-vh6bx pod in namespace openshift-monitoring (previous: false). I0507 14:45:33.087910 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-state-metrics container kube-state-metrics-7c87d874d5-k59gk pod in namespace openshift-monitoring (previous: false). W0507 14:45:33.255573 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0507 14:45:33.278812 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-main container kube-state-metrics-7c87d874d5-k59gk pod in namespace openshift-monitoring (previous: false). I0507 14:45:33.478888 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-self container kube-state-metrics-7c87d874d5-k59gk pod in namespace openshift-monitoring (previous: false). I0507 14:45:33.678363 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for metrics-server container metrics-server-7964cb5654-r5ttt pod in namespace openshift-monitoring (previous: false). I0507 14:45:33.877910 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for metrics-server container metrics-server-7964cb5654-z7kjz pod in namespace openshift-monitoring (previous: false). I0507 14:45:34.079185 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for monitoring-plugin container monitoring-plugin-67788f859f-8gxgj pod in namespace openshift-monitoring (previous: false). W0507 14:45:34.256034 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0507 14:45:34.256061 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0507 14:45:34.256078 1 tasks_processing.go:74] worker 52 stopped. E0507 14:45:34.256088 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0507 14:45:34.256100 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0507 14:45:34.256111 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.054978703s to process 1 records I0507 14:45:34.279150 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for monitoring-plugin container monitoring-plugin-67788f859f-f6ffr pod in namespace openshift-monitoring (previous: false). I0507 14:45:34.484573 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for init-textfile container node-exporter-6flvl pod in namespace openshift-monitoring (previous: false). I0507 14:45:34.701811 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 14:45:34.701832 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-exporter container node-exporter-6flvl pod in namespace openshift-monitoring (previous: false). I0507 14:45:34.879437 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container node-exporter-6flvl pod in namespace openshift-monitoring (previous: false). I0507 14:45:35.079200 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for init-textfile container node-exporter-kf7wk pod in namespace openshift-monitoring (previous: false). I0507 14:45:35.278778 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 14:45:35.278796 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-exporter container node-exporter-kf7wk pod in namespace openshift-monitoring (previous: false). I0507 14:45:35.479713 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container node-exporter-kf7wk pod in namespace openshift-monitoring (previous: false). I0507 14:45:35.678517 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for init-textfile container node-exporter-vkfw8 pod in namespace openshift-monitoring (previous: false). I0507 14:45:35.878598 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 14:45:35.878615 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-exporter container node-exporter-vkfw8 pod in namespace openshift-monitoring (previous: false). I0507 14:45:36.079616 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container node-exporter-vkfw8 pod in namespace openshift-monitoring (previous: false). I0507 14:45:36.279199 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-main container openshift-state-metrics-6cb79d4dbd-rzsd2 pod in namespace openshift-monitoring (previous: false). I0507 14:45:36.478199 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-self container openshift-state-metrics-6cb79d4dbd-rzsd2 pod in namespace openshift-monitoring (previous: false). I0507 14:45:36.678805 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for openshift-state-metrics container openshift-state-metrics-6cb79d4dbd-rzsd2 pod in namespace openshift-monitoring (previous: false). I0507 14:45:36.878583 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for init-config-reloader container prometheus-k8s-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:37.079467 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for prometheus container prometheus-k8s-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:37.280569 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for config-reloader container prometheus-k8s-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:37.479520 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for thanos-sidecar container prometheus-k8s-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:37.678786 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-web container prometheus-k8s-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:37.878593 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container prometheus-k8s-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:38.078691 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-thanos container prometheus-k8s-0 pod in namespace openshift-monitoring (previous: false). I0507 14:45:38.278931 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for init-config-reloader container prometheus-k8s-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:38.479557 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for prometheus container prometheus-k8s-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:38.680929 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for config-reloader container prometheus-k8s-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:38.878846 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for thanos-sidecar container prometheus-k8s-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:39.079522 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-web container prometheus-k8s-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:39.281003 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container prometheus-k8s-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:39.479301 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-thanos container prometheus-k8s-1 pod in namespace openshift-monitoring (previous: false). I0507 14:45:39.678511 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for prometheus-operator container prometheus-operator-5df5b8bf6d-kh8rp pod in namespace openshift-monitoring (previous: false). I0507 14:45:39.880240 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container prometheus-operator-5df5b8bf6d-kh8rp pod in namespace openshift-monitoring (previous: false). I0507 14:45:40.080040 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for prometheus-operator-admission-webhook container prometheus-operator-admission-webhook-68c48fd977-b7hml pod in namespace openshift-monitoring (previous: false). I0507 14:45:40.280463 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for prometheus-operator-admission-webhook container prometheus-operator-admission-webhook-68c48fd977-v8p5k pod in namespace openshift-monitoring (previous: false). I0507 14:45:40.479084 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for thanos-query container thanos-querier-5fdd4b94cd-pf79x pod in namespace openshift-monitoring (previous: false). I0507 14:45:40.679529 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-web container thanos-querier-5fdd4b94cd-pf79x pod in namespace openshift-monitoring (previous: false). I0507 14:45:40.880579 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container thanos-querier-5fdd4b94cd-pf79x pod in namespace openshift-monitoring (previous: false). I0507 14:45:41.078979 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for prom-label-proxy container thanos-querier-5fdd4b94cd-pf79x pod in namespace openshift-monitoring (previous: false). I0507 14:45:41.278768 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-rules container thanos-querier-5fdd4b94cd-pf79x pod in namespace openshift-monitoring (previous: false). I0507 14:45:41.481334 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-metrics container thanos-querier-5fdd4b94cd-pf79x pod in namespace openshift-monitoring (previous: false). I0507 14:45:41.677999 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for thanos-query container thanos-querier-5fdd4b94cd-tblwc pod in namespace openshift-monitoring (previous: false). I0507 14:45:41.879938 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-web container thanos-querier-5fdd4b94cd-tblwc pod in namespace openshift-monitoring (previous: false). I0507 14:45:42.062691 1 tasks_processing.go:74] worker 7 stopped. I0507 14:45:42.062739 1 recorder.go:75] Recording config/installplans with fingerprint=95dfd5f33a9a46199239158a0ec0183b629818eca24ae41f4d706d7a3d604aec I0507 14:45:42.062750 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.859949792s to process 1 records I0507 14:45:42.078788 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container thanos-querier-5fdd4b94cd-tblwc pod in namespace openshift-monitoring (previous: false). I0507 14:45:42.280090 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for prom-label-proxy container thanos-querier-5fdd4b94cd-tblwc pod in namespace openshift-monitoring (previous: false). I0507 14:45:42.480565 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-rules container thanos-querier-5fdd4b94cd-tblwc pod in namespace openshift-monitoring (previous: false). I0507 14:45:42.679618 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy-metrics container thanos-querier-5fdd4b94cd-tblwc pod in namespace openshift-monitoring (previous: false). I0507 14:45:42.808291 1 tasks_processing.go:74] worker 16 stopped. I0507 14:45:42.808587 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=f5738032f28aea743b389898fadee779c5cd56b79a04dcc121fb2c80f0e9fc94 I0507 14:45:42.808604 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.605743592s to process 1 records I0507 14:45:42.879842 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for networking-console-plugin container networking-console-plugin-6ddbfdf749-2fc88 pod in namespace openshift-network-console (previous: false). I0507 14:45:43.078589 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for networking-console-plugin container networking-console-plugin-6ddbfdf749-hppbq pod in namespace openshift-network-console (previous: false). I0507 14:45:43.284423 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console-operator container console-operator-575cd97545-twnk6 pod in namespace openshift-console-operator (previous: true). I0507 14:45:43.478004 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console-operator container console-operator-575cd97545-twnk6 pod in namespace openshift-console-operator (previous: false). I0507 14:45:43.680732 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console container console-69f7bcf9d-nhjh5 pod in namespace openshift-console (previous: false). I0507 14:45:43.879408 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console container console-69f7bcf9d-xzlc9 pod in namespace openshift-console (previous: false). I0507 14:45:44.082775 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for console container console-7cf64cc7bd-84qwk pod in namespace openshift-console (previous: false). I0507 14:45:44.279645 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for download-server container downloads-56cb7c9577-dk6xk pod in namespace openshift-console (previous: false). I0507 14:45:44.479144 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for download-server container downloads-56cb7c9577-s9fnf pod in namespace openshift-console (previous: false). I0507 14:45:44.679476 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-fqqjk pod in namespace openshift-dns (previous: false). I0507 14:45:44.881643 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-fqqjk pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-fqqjk\" is waiting to start: ContainerCreating" I0507 14:45:44.881661 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-fqqjk\" is waiting to start: ContainerCreating" I0507 14:45:44.881670 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-fqqjk pod in namespace openshift-dns (previous: false). I0507 14:45:45.082198 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-fqqjk pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-fqqjk\" is waiting to start: ContainerCreating" I0507 14:45:45.082217 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-fqqjk\" is waiting to start: ContainerCreating" I0507 14:45:45.082249 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-htx5s pod in namespace openshift-dns (previous: false). I0507 14:45:45.278130 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-htx5s pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-htx5s\" is waiting to start: ContainerCreating" I0507 14:45:45.278151 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-htx5s\" is waiting to start: ContainerCreating" I0507 14:45:45.278160 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-htx5s pod in namespace openshift-dns (previous: false). I0507 14:45:45.481788 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-htx5s pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-htx5s\" is waiting to start: ContainerCreating" I0507 14:45:45.481809 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-htx5s\" is waiting to start: ContainerCreating" I0507 14:45:45.481828 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-xv6v5 pod in namespace openshift-dns (previous: false). I0507 14:45:45.679631 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-xv6v5 pod in namespace openshift-dns (previous: false). I0507 14:45:45.878849 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-5xfm7 pod in namespace openshift-dns (previous: false). I0507 14:45:46.078162 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-l5929 pod in namespace openshift-dns (previous: false). I0507 14:45:46.278748 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 14:45:46.278769 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-twz6p pod in namespace openshift-dns (previous: false). I0507 14:45:46.479676 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 14:45:46.479695 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-54f9d4cd55-nqdgl pod in namespace openshift-image-registry (previous: false). I0507 14:45:46.683034 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-54f9d4cd55-v7pfl pod in namespace openshift-image-registry (previous: false). I0507 14:45:46.879683 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-b4d476fd-pjbpv pod in namespace openshift-image-registry (previous: true). I0507 14:45:47.095597 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-b4d476fd-pjbpv pod in namespace openshift-image-registry (previous: false). I0507 14:45:47.287658 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-29ngp pod in namespace openshift-image-registry (previous: false). I0507 14:45:47.487347 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-b9qjl pod in namespace openshift-image-registry (previous: false). I0507 14:45:47.682690 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0507 14:45:47.682712 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-mj2dg pod in namespace openshift-image-registry (previous: false). I0507 14:45:47.879530 1 tasks_processing.go:74] worker 11 stopped. I0507 14:45:47.880307 1 recorder.go:75] Recording events/openshift-monitoring with fingerprint=03f1a64702624e73ebbbbbcb12f05bf553636245988cb998cc89a76c0e16f2bc I0507 14:45:47.880353 1 recorder.go:75] Recording events/openshift-network-console with fingerprint=e2a3919aaebdb3f13c625fa6571bf7ec4d15f69c692a5c546c0b5fff04abcea7 I0507 14:45:47.880509 1 recorder.go:75] Recording events/openshift-console-operator with fingerprint=3e6d70759f11391b164e5e9226dd2418c3503a8e0f05ffa245787f4d8aa25922 I0507 14:45:47.880600 1 recorder.go:75] Recording events/openshift-console with fingerprint=443461cc1a9068ac94007db9f8c005594cb197097b3048de875ebfde3b9b166e I0507 14:45:47.880624 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=2ea21cca88b6e1c423952c4771abfc0de6930ecc2bc365a399ed4390a5599c49 I0507 14:45:47.880683 1 recorder.go:75] Recording events/openshift-dns with fingerprint=a6f19247414f6cd840270ffca946375e2c6c2cac8a83c39804e51094f21d1360 I0507 14:45:47.880812 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=72324bdefe3039b02d97a1895f2d4b74c8676b29cf8de80cf4f977e040aa682f I0507 14:45:47.880849 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-0/init-config-reloader_current.log with fingerprint=55e689c8aeb6a922c2c730be96368c7fc476a5dac334bc2649bd9ed4fbab913f I0507 14:45:47.880873 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-0/alertmanager_current.log with fingerprint=390d83514788e58e87a8b671e0c1c86ee6df59a36baab3bc0baa4557c424b80c I0507 14:45:47.880888 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-0/config-reloader_current.log with fingerprint=033d1bbfff2eb64f88ae0d7148c9636af9d3ba8ac8077d1421f6e0c816ad1bd2 I0507 14:45:47.880896 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-0/kube-rbac-proxy-web_current.log with fingerprint=f1dbd3236bbfce5869ca43ecbbd79d8767db61a8e8fff2460e6fd1db4229f7ab I0507 14:45:47.880902 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-0/kube-rbac-proxy_current.log with fingerprint=e266a34964863e841f0c1f28f6c8ddd8d8dcac2c6855441b14508caeab5e3cae I0507 14:45:47.880909 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-0/kube-rbac-proxy-metric_current.log with fingerprint=0e506050475cdd8e37ba0ebd335a1ae6eb8f772b84d74dc1954770e12acb9dfe I0507 14:45:47.880914 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-0/prom-label-proxy_current.log with fingerprint=3c98a02014b29adfe21b14fb4d9f69066b5b543c5031668538d26d870f25275a I0507 14:45:47.880920 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-1/init-config-reloader_current.log with fingerprint=01b90c8d42b672cd8a97781d09740f6a1351989681bf372496162e9db81aa197 I0507 14:45:47.880940 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-1/alertmanager_current.log with fingerprint=eeeb57d8699147f26f7e472ab21cba378e3dee42b4a60f562de327b2a95e03bf I0507 14:45:47.880954 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-1/config-reloader_current.log with fingerprint=77069d0239f91f4098495d218cc75150e86af6ad35d4cdea00f176a9a693f5c4 I0507 14:45:47.880961 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-1/kube-rbac-proxy-web_current.log with fingerprint=90fcb5d01fb3fd2e1ab589bb1699c9e2505a485a5cf3ff44671ae8224b5134bb I0507 14:45:47.880967 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-1/kube-rbac-proxy_current.log with fingerprint=1beddd937a46bbf9421898939bd94f6a56a218c688c4e67b15287172a17fd88c I0507 14:45:47.880972 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-1/kube-rbac-proxy-metric_current.log with fingerprint=2189c9175f9bf9021bf46ea8c9d247515e898da1a0e6cf918111ed0bfc89eed7 I0507 14:45:47.880976 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/alertmanager-main-1/prom-label-proxy_current.log with fingerprint=81b4ab2c164c5baccf59168b0d641000a5b541c3a62c33e7da75a911e7242443 I0507 14:45:47.881096 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/cluster-monitoring-operator-698978dbbf-vh6bx/cluster-monitoring-operator_current.log with fingerprint=7ff8cc7f5f71e2721c9195029b3d74127844219a01a460808722f9d8c76610c0 I0507 14:45:47.881116 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/kube-state-metrics-7c87d874d5-k59gk/kube-state-metrics_current.log with fingerprint=368eafb74d7f6f638a081b955bc935b519ba1778d78a0c2dbe6bc8f69abae4ee I0507 14:45:47.881125 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/kube-state-metrics-7c87d874d5-k59gk/kube-rbac-proxy-main_current.log with fingerprint=0e132672f045e7ace3bd518b6a55932e9cce57776219ad4ae3e4a0f2551dc56e I0507 14:45:47.881131 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/kube-state-metrics-7c87d874d5-k59gk/kube-rbac-proxy-self_current.log with fingerprint=a05882f0117d3bf0eb87d919606a47f640ecdaca5668bfc7f0c76fb3f0efb912 I0507 14:45:47.881139 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/metrics-server-7964cb5654-r5ttt/metrics-server_current.log with fingerprint=92db696a3d70d6653609825901012eddecd616648554714b8f933771652e9e80 I0507 14:45:47.881146 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/metrics-server-7964cb5654-z7kjz/metrics-server_current.log with fingerprint=f3f38c5a391c7ed80f66331e5ad515680ad3e330e2d1b4ede91a63fde28940d2 I0507 14:45:47.881156 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/monitoring-plugin-67788f859f-8gxgj/monitoring-plugin_current.log with fingerprint=5a79c8c2f5760eb680241c0a1fd1b4764050e61a36fd2762d143c00ceff1823d I0507 14:45:47.881168 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/monitoring-plugin-67788f859f-f6ffr/monitoring-plugin_current.log with fingerprint=0756f36566828f2aa6a746fcaf491e68339933f3fc4019147638e00e7c50c19b I0507 14:45:47.881199 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/node-exporter-6flvl/node-exporter_current.log with fingerprint=8f545ea2210a8a75289117ac5cf922190a81726340b29bbd984290242f0b2880 I0507 14:45:47.881208 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/node-exporter-6flvl/kube-rbac-proxy_current.log with fingerprint=23844113e6c88c02cac23f291e1d2bf07c47255d57ab3b3a2b63ce944f959f19 I0507 14:45:47.881234 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/node-exporter-kf7wk/node-exporter_current.log with fingerprint=34a699e476ceea63df744cd1659cceb4d537f5248ea4d9e3c8ad34aded45a565 I0507 14:45:47.881243 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/node-exporter-kf7wk/kube-rbac-proxy_current.log with fingerprint=d1701b729904f5a7e343d5e5b6a372ad89a889c892af25644029f234a230edd9 I0507 14:45:47.881269 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/node-exporter-vkfw8/node-exporter_current.log with fingerprint=04047b763545c3fad0fd2e5cfaaa68308da7a50230704220eb884312c2a5e59a I0507 14:45:47.881277 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/node-exporter-vkfw8/kube-rbac-proxy_current.log with fingerprint=0d68eb406fab80fb684702680f6ee019d63e325a9042edfce716b04f58ee3969 I0507 14:45:47.881283 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/openshift-state-metrics-6cb79d4dbd-rzsd2/kube-rbac-proxy-main_current.log with fingerprint=5964aadaebe7e042e74d1f83ba7d120bc1b0b1b4d540be96ac218f6fd0556836 I0507 14:45:47.881289 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/openshift-state-metrics-6cb79d4dbd-rzsd2/kube-rbac-proxy-self_current.log with fingerprint=284b075db5bab76b4b4d4843bee48dc859896a9de9bcf2eb3209c3b5e0ca0a5a I0507 14:45:47.881299 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/openshift-state-metrics-6cb79d4dbd-rzsd2/openshift-state-metrics_current.log with fingerprint=926e64da5627a8224154a47cccce1d29b139a40b1c2dfce7dd5421ef575d86dc I0507 14:45:47.881308 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-0/init-config-reloader_current.log with fingerprint=582f42e4b10fc388619c63c6d9313c1de4f0004e0b73bdf1a79b89e2a58d18b7 I0507 14:45:47.881416 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-0/prometheus_current.log with fingerprint=1275e8d19c680bea8921fdb1d9df4e73d5c6d8acc0883601f0b35a30ac8cff6e I0507 14:45:47.881430 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-0/config-reloader_current.log with fingerprint=195f055cbfe2b4df80917ab9aca85bd33cf88f8e3bd360a7b7b31e7e14d77287 I0507 14:45:47.881443 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-0/thanos-sidecar_current.log with fingerprint=494e4df765247bdc350d9abf17b3da782ffa095eecba48dbbe50ae19a49c7af4 I0507 14:45:47.881450 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-0/kube-rbac-proxy-web_current.log with fingerprint=49f48ece2d353f13cf7aeaae185ff3fbcd75a760dc1c8795f6a39c928aa67f66 I0507 14:45:47.881456 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-0/kube-rbac-proxy_current.log with fingerprint=572f6b5fdb986f4508a1b7bbf8b7a9866b99534cbf377fec219b9c49c9b40511 I0507 14:45:47.881464 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-0/kube-rbac-proxy-thanos_current.log with fingerprint=3da641eff8a5e3353caac4ffa3ad3c03a68f5c4317f65d87e10dd65f9594f1bd I0507 14:45:47.881470 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-1/init-config-reloader_current.log with fingerprint=d1eb16c14783d62546b1dd0617106905329ec07343bcc9859aa104c4da13e509 I0507 14:45:47.881567 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-1/prometheus_current.log with fingerprint=d1612a6bcc2894d19e72a6089f2fc0d7c471e3a78b32cc7525a02edb14e4e5e0 I0507 14:45:47.881581 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-1/config-reloader_current.log with fingerprint=0d664b31ba0ce9f64c4b892d8a1b594e5c9dd3284ed93ba7581da928b948a582 I0507 14:45:47.881593 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-1/thanos-sidecar_current.log with fingerprint=1256504658da16dc37b3b96a5c63c75a24844e5750905879aaa6ed2fb9925d6c I0507 14:45:47.881601 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-1/kube-rbac-proxy-web_current.log with fingerprint=37d551bfb209007240281a5157fcc94b7ddaa3bad889425a48dc7dc62b76ad75 I0507 14:45:47.881607 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-1/kube-rbac-proxy_current.log with fingerprint=bd279564edfe0d140da44b551a4488abd99a0215fd35689ab9277ff985d0ccd8 I0507 14:45:47.881613 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-k8s-1/kube-rbac-proxy-thanos_current.log with fingerprint=c5efd19602295f82d75c3c1d49a00c8c76e93f388507e2cf6a3353a8ceda0cc3 I0507 14:45:47.881692 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-operator-5df5b8bf6d-kh8rp/prometheus-operator_current.log with fingerprint=2fd26966f0e1822255b18d0f5d274f369737fcf9768fbec3878a02a6fd753482 I0507 14:45:47.881700 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-operator-5df5b8bf6d-kh8rp/kube-rbac-proxy_current.log with fingerprint=dce68c75bb2d3e0194282547c7545f9f868a5a8eb367e2879d3bb8680f5a3739 I0507 14:45:47.881708 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-operator-admission-webhook-68c48fd977-b7hml/prometheus-operator-admission-webhook_current.log with fingerprint=bbc71fd7d24722e23ea4f2488c7f1d8ae95ce1bcb5f37e70ead84515122c3478 I0507 14:45:47.881717 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/prometheus-operator-admission-webhook-68c48fd977-v8p5k/prometheus-operator-admission-webhook_current.log with fingerprint=6cab1051c876f3c922a245cf22dd99a2860bd7afa77ad28d885118a90d6218c3 I0507 14:45:47.881740 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-pf79x/thanos-query_current.log with fingerprint=50d47266ff49c464cbf4ce980a7c8569feb6b3eb9c8e713b958e6dcd30474125 I0507 14:45:47.881753 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-pf79x/kube-rbac-proxy-web_current.log with fingerprint=a20d473bc94470ac18a051f1fcaa9896afc88f8b005d5c0762627bc4a90a22a7 I0507 14:45:47.881759 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-pf79x/kube-rbac-proxy_current.log with fingerprint=4d82b66ab7a5a4b6adf6ce6e35db20989609aff1c2f347b44b981d98a9ebcced I0507 14:45:47.881764 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-pf79x/prom-label-proxy_current.log with fingerprint=d624c5bcfea0a1975db0e8b37e7364998ac6dfdf83822f52c81315fc46c62ad5 I0507 14:45:47.881769 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-pf79x/kube-rbac-proxy-rules_current.log with fingerprint=c31f665f087c22bb21896f69e2c1c285628e52561774a52a95b1d6f9558eebc4 I0507 14:45:47.881777 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-pf79x/kube-rbac-proxy-metrics_current.log with fingerprint=6b5aa8dc163aae876518fe79630c29b5af3e600cd5f3193c05506e22aedfe79e I0507 14:45:47.881800 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-tblwc/thanos-query_current.log with fingerprint=8ab256e22423d2838fc44513b0035e8a3925986631849b5e5fefa5dd04810899 I0507 14:45:47.881812 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-tblwc/kube-rbac-proxy-web_current.log with fingerprint=031dfd034a0be81a1d1b158a8be7111f2aa94d86e5d2a7eae99296d3c5bfce87 I0507 14:45:47.881817 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-tblwc/kube-rbac-proxy_current.log with fingerprint=72d70d25985f39964b225f1f530929025785878bddda0417b8dfd7efc9374330 I0507 14:45:47.881821 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-tblwc/prom-label-proxy_current.log with fingerprint=3ef36fb224e55028fac334dfef3c5fcc94c075cf5e3e2311123e7cc97a399697 I0507 14:45:47.881827 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-tblwc/kube-rbac-proxy-rules_current.log with fingerprint=fb7589cf58ca06812f41ac5f9d51431209950bdcdcdfe91f4c00a7274391228a I0507 14:45:47.881833 1 recorder.go:75] Recording config/pod/openshift-monitoring/logs/thanos-querier-5fdd4b94cd-tblwc/kube-rbac-proxy-metrics_current.log with fingerprint=596959456563438fabb4fb37ef390c825ae670f6b2d279619a91aed74d118931 I0507 14:45:47.881842 1 recorder.go:75] Recording config/pod/openshift-network-console/logs/networking-console-plugin-6ddbfdf749-2fc88/networking-console-plugin_current.log with fingerprint=4426ee30844201879d44fbeb0e66d25d2967751677f2a2edc03a73c86f4fb0d5 I0507 14:45:47.881847 1 recorder.go:75] Recording config/pod/openshift-network-console/logs/networking-console-plugin-6ddbfdf749-hppbq/networking-console-plugin_current.log with fingerprint=44b4b7f11d372a0054a04104d15eac8edcdb0fec2ea1036c71b01ec74ddaac5d I0507 14:45:47.881980 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-twnk6 with fingerprint=9722d88e02f17dfdf2c67b0816ce8e77157422af099de5e27f722e931c8794a2 E0507 14:45:47.881999 1 gather.go:161] error recording gatherer "clusterconfig" function "operators_pods_and_events" result "config/pod/openshift-console-operator/console-operator-575cd97545-twnk6.json" because of the error: the record with the same name "config/pod/openshift-console-operator/console-operator-575cd97545-twnk6.json" was already recorded and had the fingerprint "9722d88e02f17dfdf2c67b0816ce8e77157422af099de5e27f722e931c8794a2", overwriting with the record having fingerprint "9722d88e02f17dfdf2c67b0816ce8e77157422af099de5e27f722e931c8794a2" W0507 14:45:47.882013 1 gather.go:155] issue recording gatherer "clusterconfig" function "operators_pods_and_events" result "config/pod/openshift-console-operator/console-operator-575cd97545-twnk6.json" because of the warning: warning: the record with the same fingerprint "9722d88e02f17dfdf2c67b0816ce8e77157422af099de5e27f722e931c8794a2" was already recorded at path "config/pod/openshift-console-operator/console-operator-575cd97545-twnk6.json", recording another one with a different path "config/pod/openshift-console-operator/console-operator-575cd97545-twnk6.json" I0507 14:45:47.882026 1 recorder.go:75] Recording config/pod/openshift-console-operator/logs/console-operator-575cd97545-twnk6/console-operator_previous.log with fingerprint=be0c34f71489c24606af24dc5e4df6ab0003357ff0b771f8d85317628b70bab5 I0507 14:45:47.882172 1 recorder.go:75] Recording config/pod/openshift-console-operator/logs/console-operator-575cd97545-twnk6/console-operator_current.log with fingerprint=a07bdcb1ad32054c40cc79e00c9d6210dd72c2476417f597c96366be592ade94 I0507 14:45:47.882188 1 recorder.go:75] Recording config/pod/openshift-console/logs/console-69f7bcf9d-nhjh5/console_current.log with fingerprint=bf5d230cc3a25405c4794fd59d96552a2e4ff5080e97744081539e1ac3d063a1 I0507 14:45:47.882203 1 recorder.go:75] Recording config/pod/openshift-console/logs/console-69f7bcf9d-xzlc9/console_current.log with fingerprint=772d1f391767f2cc38a2c044bad730b175e9e570835bd7f1d32351253db0ada6 I0507 14:45:47.882218 1 recorder.go:75] Recording config/pod/openshift-console/logs/console-7cf64cc7bd-84qwk/console_current.log with fingerprint=4276e445facd7069300cb139e6867e54ee45641007c1c9aba3c20a69188f2f40 I0507 14:45:47.882224 1 recorder.go:75] Recording config/pod/openshift-console/logs/downloads-56cb7c9577-dk6xk/download-server_current.log with fingerprint=11d2b0f75e5c9de04cb5c918e363fc489e8ee0b97a01635154c51459c1f6a35f I0507 14:45:47.882230 1 recorder.go:75] Recording config/pod/openshift-console/logs/downloads-56cb7c9577-s9fnf/download-server_current.log with fingerprint=2d8359f29a6603792600398bedd4f660ee1e397f4d7d01135ee1f0dd3f95f239 I0507 14:45:47.882317 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-fqqjk with fingerprint=649b3bf368bc65b00eb41420226dc93aa2cdeee3f4e8e21efd29f479f88bb8c8 I0507 14:45:47.882402 1 recorder.go:75] Recording config/pod/openshift-dns/dns-default-htx5s with fingerprint=62c46f15d4f85c51142809bd3e67e6d1a0c7ff31880a830c97826fea2d2a852b I0507 14:45:47.882411 1 recorder.go:75] Recording config/pod/openshift-dns/logs/dns-default-xv6v5/dns_current.log with fingerprint=bed7c338ee0ec89d5a6ffd5a42b4a23016f2e74944955b241887931936283533 I0507 14:45:47.882418 1 recorder.go:75] Recording config/pod/openshift-dns/logs/dns-default-xv6v5/kube-rbac-proxy_current.log with fingerprint=c0ede67707ef48f151abb990dd3e0f197f1cde69b111ed2e3dc65e1a81efe27b I0507 14:45:47.882422 1 recorder.go:75] Recording config/pod/openshift-dns/logs/node-resolver-5xfm7/dns-node-resolver_current.log with fingerprint=3b59b912d78152bd7dd397c4a2fc6e84bd35d0f416c13bc64d47f18dec38034f I0507 14:45:47.882447 1 recorder.go:75] Recording config/pod/openshift-image-registry/logs/image-registry-54f9d4cd55-nqdgl/registry_current.log with fingerprint=94782c858904c5bc1fef44dd1de1b702f54cb2167d61437a2f106f36aca5e267 I0507 14:45:47.882465 1 recorder.go:75] Recording config/pod/openshift-image-registry/logs/image-registry-54f9d4cd55-v7pfl/registry_current.log with fingerprint=b49ec83221d77d3a5631db90ae03361803524f53593872d6404881a718f40fbc I0507 14:45:47.882592 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-b4d476fd-pjbpv with fingerprint=60157c47e85e3a67f96b65a7984b8a0759ad783d8e3a27f8c1de2e9696aa68cb I0507 14:45:47.882638 1 recorder.go:75] Recording config/pod/openshift-image-registry/logs/image-registry-b4d476fd-pjbpv/registry_previous.log with fingerprint=27f16fd3693d112a2bbc962046ea3624c7eb14f08c233c85fc24bb2f09554b02 I0507 14:45:47.882663 1 recorder.go:75] Recording config/pod/openshift-image-registry/logs/image-registry-b4d476fd-pjbpv/registry_current.log with fingerprint=5419957b739bacc89a76a410f9871a1b7ff12c01e32961bcaf42985516b7b466 I0507 14:45:47.882671 1 recorder.go:75] Recording config/pod/openshift-image-registry/logs/node-ca-29ngp/node-ca_current.log with fingerprint=021c2e237f5f21fd3cc8fe50fbcdd33205f6d7abd5349344ef1e8bb203cc12ef I0507 14:45:47.882678 1 recorder.go:75] Recording config/pod/openshift-image-registry/logs/node-ca-mj2dg/node-ca_current.log with fingerprint=5f25a7e72ae520b6a6ceb38b4a643c79084825a94abe47330db13ade95edf72e I0507 14:45:47.882684 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 18.679934301s to process 90 records E0507 14:45:47.882745 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 18.684s with: function \"pod_network_connectivity_checks\" failed with an error, function \"tsdb_status\" failed with an error, function \"metrics\" failed with an error, function \"machines\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error, unable to record function \"operators_pods_and_events\" record \"config/pod/openshift-console-operator/console-operator-575cd97545-twnk6.json\"" I0507 14:45:47.883854 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "tsdb_status" failed with an error, function "metrics" failed with an error, function "machines" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error, unable to record function "operators_pods_and_events" record "config/pod/openshift-console-operator/console-operator-575cd97545-twnk6.json" I0507 14:45:47.883869 1 periodic.go:209] Running workloads gatherer I0507 14:45:47.883884 1 tasks_processing.go:45] number of workers: 2 I0507 14:45:47.883892 1 tasks_processing.go:69] worker 1 listening for tasks. I0507 14:45:47.883896 1 tasks_processing.go:71] worker 1 working on helmchart_info task. I0507 14:45:47.883904 1 tasks_processing.go:69] worker 0 listening for tasks. I0507 14:45:47.883981 1 tasks_processing.go:71] worker 0 working on workload_info task. I0507 14:45:47.890351 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-2-38.ec2.internal... I0507 14:45:47.890358 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-1-217.ec2.internal... I0507 14:45:47.890381 1 gather_workloads_runtime_infos.go:63] Gathering workload runtime info for node ip-10-0-0-220.ec2.internal... I0507 14:45:47.910255 1 tasks_processing.go:74] worker 1 stopped. I0507 14:45:47.910277 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 26.342319ms to process 0 records I0507 14:45:54.846013 1 gather_workloads_runtime_infos.go:84] Gathered workload runtime infos in 7s I0507 14:45:54.872635 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 25s for image data I0507 14:45:54.881867 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (10ms) I0507 14:45:54.891543 1 gather_workloads_info.go:387] No image sha256:04c87c054a3f366a7dfbe0a93ebb0c80a098ee16842c5794b67c1202eec61996 (10ms) I0507 14:45:54.900421 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (9ms) I0507 14:45:54.909483 1 gather_workloads_info.go:387] No image sha256:0092c6ff139f8698c91349d507eb3182bc5d6c4ae040f672e347ff504accdea2 (9ms) I0507 14:45:54.919196 1 gather_workloads_info.go:387] No image sha256:6e5df2aab9ce759e8f957341a11c087c233307e3c8f8760d21270efc19ccd3ce (10ms) I0507 14:45:54.928220 1 gather_workloads_info.go:387] No image sha256:0af73ad7a47c9ff53269ae2bc5163bf9e389085220c34e4bd42c92c4e7a85f6d (9ms) I0507 14:45:54.937107 1 gather_workloads_info.go:387] No image sha256:1e65f9f5490fbc930d2dce96f03ddcb28990b839d7dc011ef5de5bec6c5095bf (9ms) I0507 14:45:54.946315 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (9ms) I0507 14:45:54.955173 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (9ms) I0507 14:45:54.964460 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (9ms) I0507 14:45:54.981201 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (17ms) I0507 14:45:55.082050 1 gather_workloads_info.go:387] No image sha256:b99613494e9d6a4a8207323872d63cc02b830fc80a8bf972e62c64381aa4cd17 (101ms) I0507 14:45:55.181277 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (99ms) I0507 14:45:55.281812 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (101ms) I0507 14:45:55.381578 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (100ms) I0507 14:45:55.481709 1 gather_workloads_info.go:387] No image sha256:513b1166e376feca472fdd5174f6abf0abf02925d6a21b7b4d89a2ff2fc67bed (100ms) I0507 14:45:55.581676 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (100ms) I0507 14:45:55.681620 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0507 14:45:55.781736 1 gather_workloads_info.go:387] No image sha256:76cc47f3775560477ec548c7b6fddbd99112ad6604f790697eef546576051fb7 (100ms) I0507 14:45:55.882281 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (101ms) I0507 14:45:55.981758 1 gather_workloads_info.go:387] No image sha256:25bda9d34e23ba6bc9df5cf7104bdb237269740a3a76ee025f88b34e68b7e2b5 (99ms) I0507 14:45:56.081437 1 gather_workloads_info.go:387] No image sha256:1477f3287cc75352305274f248f781c1b7c2a8696f4f7bc450c45a4a5f5b10ea (100ms) I0507 14:45:56.181478 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (100ms) I0507 14:45:56.281854 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (100ms) I0507 14:45:56.383939 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (102ms) I0507 14:45:56.484632 1 gather_workloads_info.go:387] No image sha256:7bd3e88d6d3f878a89203df590b684c5a77bf08df4717451d3197ab74683a80e (101ms) I0507 14:45:56.582176 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (98ms) I0507 14:45:56.681674 1 gather_workloads_info.go:387] No image sha256:32d4561d4bf1b3c4e3a7878f918f9c1422bab85bdf8711fbdf905d59692fee81 (99ms) I0507 14:45:56.781811 1 gather_workloads_info.go:387] No image sha256:3d6ea9e1633f38070718a1357fcb80e6cb3932aafcc4d65e659795b186c5da15 (100ms) I0507 14:45:56.888696 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (107ms) I0507 14:45:56.982978 1 gather_workloads_info.go:387] No image sha256:74713675d580a47a56b628469c4c05c41e114e0b935960d718ff0cf7e72f613d (94ms) I0507 14:45:57.083097 1 gather_workloads_info.go:387] No image sha256:90bdf9e678fe18e35401e34e1cb0912d822e9cb32b9e41cc545870540e9a87f2 (100ms) I0507 14:45:57.181660 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (99ms) I0507 14:45:57.285216 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (104ms) I0507 14:45:57.382685 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (97ms) I0507 14:45:57.482131 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (99ms) I0507 14:45:57.582453 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (100ms) I0507 14:45:57.682528 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (100ms) I0507 14:45:57.781677 1 gather_workloads_info.go:387] No image sha256:101993f6e65d6fda8b90c583b718b720809182961f69bc777295f47e6df4d7ce (99ms) I0507 14:45:57.884645 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (103ms) I0507 14:45:57.981557 1 gather_workloads_info.go:387] No image sha256:50197f22710766515f67944a779e00dd9ae3d17b18732d7324a970353b11f292 (97ms) I0507 14:45:58.081680 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (100ms) I0507 14:45:58.182177 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (100ms) I0507 14:45:58.282575 1 gather_workloads_info.go:398] Found image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (100ms) I0507 14:45:58.382105 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (99ms) I0507 14:45:58.482288 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (100ms) I0507 14:45:58.581244 1 gather_workloads_info.go:387] No image sha256:3612c5a433effd17b7aff7ff6edf4fb27b61153908e3a0294158e4886a1d600d (99ms) I0507 14:45:58.581285 1 tasks_processing.go:74] worker 0 stopped. E0507 14:45:58.581298 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: received unexpected status code 500 Internal Server Error from https://10.128.0.15:8443/gather_runtime_info I0507 14:45:58.581611 1 recorder.go:75] Recording config/workload_info with fingerprint=83501224e144ee2fa77dc3bb230f14beb37c632349524365938b0d2d3f67e90e I0507 14:45:58.581630 1 gather.go:177] gatherer "workloads" function "workload_info" took 10.697288881s to process 1 records E0507 14:45:58.581666 1 periodic.go:247] "Unhandled Error" err="workloads failed after 10.697s with: function \"workload_info\" failed with an error" I0507 14:45:58.582772 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0507 14:45:58.582785 1 periodic.go:209] Running conditional gatherer I0507 14:45:58.588667 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0507 14:45:58.763740 1 conditional_gatherer.go:338] updating alerts cache for conditional gatherer I0507 14:45:59.827433 1 conditional_gatherer.go:374] alert "PrometheusKubernetesListWatchFailures" has state "pending" I0507 14:45:59.827451 1 conditional_gatherer.go:374] alert "PrometheusKubernetesListWatchFailures" has state "pending" I0507 14:45:59.827455 1 conditional_gatherer.go:374] alert "TargetDown" has state "pending" I0507 14:45:59.827458 1 conditional_gatherer.go:374] alert "TargetDown" has state "pending" I0507 14:45:59.827460 1 conditional_gatherer.go:374] alert "ClusterMonitoringOperatorReconciliationErrors" has state "pending" I0507 14:45:59.827463 1 conditional_gatherer.go:374] alert "KubePodNotScheduled" has state "pending" I0507 14:45:59.827542 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0507 14:45:59.834097 1 conditional_gatherer.go:392] cluster version is '4.20.8' I0507 14:45:59.834131 1 tasks_processing.go:45] number of workers: 3 I0507 14:45:59.834140 1 tasks_processing.go:69] worker 2 listening for tasks. I0507 14:45:59.834144 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0507 14:45:59.834152 1 tasks_processing.go:69] worker 0 listening for tasks. I0507 14:45:59.834162 1 tasks_processing.go:69] worker 1 listening for tasks. I0507 14:45:59.834176 1 tasks_processing.go:74] worker 1 stopped. I0507 14:45:59.834189 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0507 14:45:59.834194 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0507 14:45:59.834244 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=c8d4892cd4f2eea27d57353c6055eeb530bc2d811bfe041f6984d90395255a8c I0507 14:45:59.834259 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.411µs to process 1 records I0507 14:45:59.834300 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=5f681cc1442447fd2c1534f876836dff54caf3c545b582a5a8aae894f9203c5d I0507 14:45:59.834308 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 948ns to process 1 records I0507 14:45:59.834313 1 tasks_processing.go:74] worker 0 stopped. I0507 14:45:59.834476 1 tasks_processing.go:74] worker 2 stopped. I0507 14:45:59.834488 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 268.623µs to process 0 records I0507 14:45:59.834504 1 controllerstatus.go:80] name=periodic-conditional healthy=true reason=Succeeded message= I0507 14:45:59.834520 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=363fa5c8fa2573b18bc685faeca9ca9f021a8f80760895b7701f54e3dcd5c85f I0507 14:45:59.859777 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=0dd81ed404c8094152f23bc6344034b334ee293a7aa21f48bf0d8ceb27e1c711 I0507 14:45:59.859930 1 diskrecorder.go:70] Writing 199 records to /var/lib/insights-operator/insights-2026-05-07-144559.tar.gz I0507 14:45:59.873544 1 diskrecorder.go:51] Wrote 199 records to disk in 13ms I0507 14:45:59.873577 1 periodic.go:278] Gathering cluster info every 2h0m0s I0507 14:45:59.873591 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0507 14:46:59.198580 1 diskrecorder.go:223] Found files to send: insights-2026-05-07-144559.tar.gz I0507 14:46:59.198612 1 insightsuploader.go:150] Checking archives to upload periodically every 15m28.365280103s I0507 14:46:59.198621 1 insightsuploader.go:165] Uploading latest report since 0001-01-01T00:00:00Z I0507 14:46:59.213496 1 requests.go:46] Uploading application/vnd.redhat.openshift.periodic to https://console.redhat.com/api/ingress/v1/upload I0507 14:46:59.502187 1 requests.go:87] Successfully reported id=2026-05-07T14:46:59Z x-rh-insights-request-id=f0e3eec1a90149bebc2f30ca45701216, wrote=102157 I0507 14:46:59.502266 1 insightsuploader.go:187] Uploaded report successfully in 303.637323ms I0507 14:46:59.502290 1 controller.go:128] Initializing last reported time to 2026-05-07T14:46:59Z I0507 14:46:59.502306 1 insightsreport.go:304] Archive uploaded, starting pulling report... I0507 14:46:59.502318 1 insightsreport.go:215] Starting retrieving report from Smart Proxy I0507 14:46:59.502326 1 insightsreport.go:221] Initial delay for pulling: 1m0s I0507 14:46:59.507355 1 controller.go:512] The operator is healthy I0507 14:47:29.236102 1 controller.go:512] The operator is healthy I0507 14:48:04.838588 1 insightsreport.go:137] Pulling report from smart-proxy I0507 14:48:04.838617 1 insightsreport.go:149] Retrieving report I0507 14:48:04.844977 1 requests.go:110] Retrieving report for cluster: 0c8fe701-428b-471f-9475-8e5eab5310e4 I0507 14:48:04.844992 1 requests.go:111] Endpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/0c8fe701-428b-471f-9475-8e5eab5310e4/reports I0507 14:48:04.847698 1 requests.go:121] Retrieving report from https://console.redhat.com/api/insights-results-aggregator/v2/cluster/0c8fe701-428b-471f-9475-8e5eab5310e4/reports I0507 14:48:05.082915 1 insightsreport.go:184] Report retrieved I0507 14:48:05.107078 1 insightsreport.go:239] Report retrieved correctly I0507 14:49:29.240469 1 controller.go:512] The operator is healthy I0507 14:50:29.192421 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0507 14:50:29.200690 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0507 14:50:29.200712 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0507 14:50:29.204368 1 secretconfigobserver.go:119] support secret does not exist I0507 14:51:29.243896 1 controller.go:512] The operator is healthy I0507 14:53:29.236924 1 controller.go:512] The operator is healthy I0507 14:55:29.184008 1 configmapobserver.go:84] configmaps "insights-config" not found I0507 14:55:29.187759 1 configmapobserver.go:84] configmaps "insights-config" not found I0507 14:55:29.204863 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0507 14:55:29.209518 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0507 14:55:29.209536 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0507 14:55:29.213337 1 secretconfigobserver.go:119] support secret does not exist I0507 14:55:29.236106 1 controller.go:512] The operator is healthy I0507 14:57:29.236830 1 controller.go:512] The operator is healthy I0507 14:59:29.236775 1 controller.go:512] The operator is healthy I0507 15:00:29.213738 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0507 15:00:29.218965 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0507 15:00:29.218986 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0507 15:00:29.222676 1 secretconfigobserver.go:119] support secret does not exist I0507 15:01:29.238612 1 controller.go:512] The operator is healthy I0507 15:02:32.868748 1 insightsuploader.go:146] Nothing to report since 2026-05-07T14:46:59Z I0507 15:03:29.241600 1 controller.go:512] The operator is healthy I0507 15:05:29.188486 1 configmapobserver.go:84] configmaps "insights-config" not found I0507 15:05:29.192352 1 configmapobserver.go:84] configmaps "insights-config" not found I0507 15:05:29.223428 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0507 15:05:29.228025 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0507 15:05:29.228051 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0507 15:05:29.231868 1 secretconfigobserver.go:119] support secret does not exist I0507 15:05:29.235769 1 controller.go:512] The operator is healthy I0507 15:07:29.236779 1 controller.go:512] The operator is healthy I0507 15:09:29.237419 1 controller.go:512] The operator is healthy I0507 15:10:29.232902 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0507 15:10:29.237430 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0507 15:10:29.237449 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0507 15:10:29.241087 1 secretconfigobserver.go:119] support secret does not exist I0507 15:11:29.243827 1 controller.go:512] The operator is healthy I0507 15:13:29.236777 1 controller.go:512] The operator is healthy I0507 15:15:29.185199 1 configmapobserver.go:84] configmaps "insights-config" not found I0507 15:15:29.235873 1 controller.go:512] The operator is healthy I0507 15:15:29.241726 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0507 15:15:29.246116 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0507 15:15:29.246134 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0507 15:15:29.250087 1 secretconfigobserver.go:119] support secret does not exist I0507 15:17:29.236764 1 controller.go:512] The operator is healthy I0507 15:17:42.588856 1 insightsuploader.go:146] Nothing to report since 2026-05-07T14:46:59Z I0507 15:19:29.236505 1 controller.go:512] The operator is healthy I0507 15:20:29.250565 1 secretconfigobserver.go:136] Refreshing configuration from cluster pull secret I0507 15:20:29.255453 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0507 15:20:29.255473 1 secretconfigobserver.go:162] Refreshing configuration from cluster support secret I0507 15:20:29.258952 1 secretconfigobserver.go:119] support secret does not exist