W0428 15:37:40.509908 1 cmd.go:257] Using insecure, self-signed certificates I0428 15:37:40.593664 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0428 15:37:40.594041 1 observer_polling.go:159] Starting file observer I0428 15:37:41.217960 1 operator.go:76] Starting insights-operator v0.0.0-master+$Format:%H$ I0428 15:37:41.218212 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0428 15:37:41.218906 1 secure_serving.go:57] Forcing use of http/1.1 only W0428 15:37:41.218934 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0428 15:37:41.218941 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. I0428 15:37:41.218939 1 simple_featuregate_reader.go:171] Starting feature-gate-detector W0428 15:37:41.218945 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0428 15:37:41.218950 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0428 15:37:41.218952 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0428 15:37:41.218955 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0428 15:37:41.225992 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0428 15:37:41.226021 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0428 15:37:41.225879 1 operator.go:141] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GCPClusterHostedDNSInstall GatewayAPI GatewayAPIController HighlyAvailableArbiter HyperShiftOnlyDynamicResourceAllocation ImageStreamImportMode ImageVolume KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS ManagedBootImagesAzure ManagedBootImagesvSphere MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages PreconfiguredUDNAddresses ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SigstoreImageVerification SigstoreImageVerificationPKI StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks VolumeAttributesClass AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSDualStackInstall AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureDualStackInstall AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement CBORServingAndStorage CRDCompatibilityRequirementOperator ClientsAllowCBOR ClientsPreferCBOR ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterAPIMachineManagement ClusterAPIMachineManagementVSphere ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud EtcdBackendQuota EventTTL EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall GCPDualStackInstall ImageModeStatusReporting IngressControllerDynamicConfigurationManager InsightsConfig InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryption KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesCPMS MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutableCSINodeAllocatableCount MutatingAdmissionPolicy NewOLM NewOLMBoxCutterRuntime NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterInstall NutanixMultiSubnets OSStreams OVNObservability OnPremDNSRecords OpenShiftPodSecurityAdmission ProvisioningRequestAvailable SELinuxMount ShortCertRotation SignatureStores TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeGroupSnapshot] I0428 15:37:41.226018 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0428 15:37:41.226047 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0428 15:37:41.226065 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0428 15:37:41.226083 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0428 15:37:41.226334 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"ca931221-db25-433d-8a0b-dc230f8ac497", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GCPClusterHostedDNSInstall", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "HyperShiftOnlyDynamicResourceAllocation", "ImageStreamImportMode", "ImageVolume", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "PreconfiguredUDNAddresses", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SigstoreImageVerification", "SigstoreImageVerificationPKI", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks", "VolumeAttributesClass"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSDualStackInstall", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureDualStackInstall", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "CBORServingAndStorage", "CRDCompatibilityRequirementOperator", "ClientsAllowCBOR", "ClientsPreferCBOR", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterAPIMachineManagement", "ClusterAPIMachineManagementVSphere", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "EtcdBackendQuota", "EventTTL", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "GCPDualStackInstall", "ImageModeStatusReporting", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryption", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesCPMS", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutableCSINodeAllocatableCount", "MutatingAdmissionPolicy", "NewOLM", "NewOLMBoxCutterRuntime", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterInstall", "NutanixMultiSubnets", "OSStreams", "OVNObservability", "OnPremDNSRecords", "OpenShiftPodSecurityAdmission", "ProvisioningRequestAvailable", "SELinuxMount", "ShortCertRotation", "SignatureStores", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeGroupSnapshot"}} I0428 15:37:41.226471 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-3927914365/tls.crt::/tmp/serving-cert-3927914365/tls.key" I0428 15:37:41.227191 1 secure_serving.go:213] Serving securely on [::]:8443 I0428 15:37:41.227222 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0428 15:37:41.244686 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0428 15:37:41.244718 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0428 15:37:41.244836 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0428 15:37:41.255399 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0428 15:37:41.255425 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0428 15:37:41.296133 1 secretconfigobserver.go:119] support secret does not exist I0428 15:37:41.302565 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0428 15:37:41.309564 1 secretconfigobserver.go:119] support secret does not exist I0428 15:37:41.313591 1 recorder.go:176] Pruning old reports every 7h58m43s, max age is 288h0m0s I0428 15:37:41.319616 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0428 15:37:41.319622 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0428 15:37:41.319644 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0428 15:37:41.319648 1 insightsreport.go:296] Starting report retriever I0428 15:37:41.319658 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0428 15:37:41.319666 1 periodic.go:216] Running clusterconfig gatherer I0428 15:37:41.319725 1 tasks_processing.go:45] number of workers: 64 I0428 15:37:41.319765 1 tasks_processing.go:69] worker 4 listening for tasks. I0428 15:37:41.319773 1 tasks_processing.go:69] worker 1 listening for tasks. I0428 15:37:41.319782 1 tasks_processing.go:69] worker 0 listening for tasks. I0428 15:37:41.319784 1 tasks_processing.go:69] worker 2 listening for tasks. I0428 15:37:41.319788 1 tasks_processing.go:69] worker 3 listening for tasks. I0428 15:37:41.319791 1 tasks_processing.go:71] worker 0 working on machine_config_pools task. I0428 15:37:41.319794 1 tasks_processing.go:71] worker 2 working on sap_config task. I0428 15:37:41.319799 1 tasks_processing.go:69] worker 19 listening for tasks. I0428 15:37:41.319801 1 tasks_processing.go:69] worker 20 listening for tasks. I0428 15:37:41.319809 1 tasks_processing.go:69] worker 5 listening for tasks. I0428 15:37:41.319812 1 tasks_processing.go:69] worker 21 listening for tasks. I0428 15:37:41.319817 1 tasks_processing.go:69] worker 6 listening for tasks. I0428 15:37:41.319821 1 tasks_processing.go:69] worker 22 listening for tasks. I0428 15:37:41.319825 1 tasks_processing.go:69] worker 7 listening for tasks. I0428 15:37:41.319829 1 tasks_processing.go:69] worker 23 listening for tasks. I0428 15:37:41.319833 1 tasks_processing.go:69] worker 8 listening for tasks. I0428 15:37:41.319838 1 tasks_processing.go:69] worker 24 listening for tasks. I0428 15:37:41.319841 1 tasks_processing.go:69] worker 9 listening for tasks. I0428 15:37:41.319847 1 tasks_processing.go:69] worker 25 listening for tasks. I0428 15:37:41.320011 1 tasks_processing.go:71] worker 25 working on operators_pods_and_events task. I0428 15:37:41.319793 1 tasks_processing.go:69] worker 34 listening for tasks. I0428 15:37:41.319849 1 tasks_processing.go:69] worker 10 listening for tasks. I0428 15:37:41.319856 1 tasks_processing.go:69] worker 11 listening for tasks. I0428 15:37:41.319864 1 tasks_processing.go:69] worker 16 listening for tasks. I0428 15:37:41.319876 1 tasks_processing.go:69] worker 15 listening for tasks. I0428 15:37:41.319874 1 tasks_processing.go:69] worker 50 listening for tasks. I0428 15:37:41.319875 1 tasks_processing.go:69] worker 17 listening for tasks. I0428 15:37:41.319887 1 tasks_processing.go:69] worker 12 listening for tasks. I0428 15:37:41.320102 1 tasks_processing.go:71] worker 17 working on ingress_certificates task. I0428 15:37:41.320112 1 tasks_processing.go:71] worker 11 working on image_pruners task. I0428 15:37:41.320125 1 tasks_processing.go:71] worker 34 working on validating_webhook_configurations task. I0428 15:37:41.320147 1 tasks_processing.go:71] worker 15 working on storage_classes task. I0428 15:37:41.320106 1 tasks_processing.go:71] worker 12 working on active_alerts task. I0428 15:37:41.320152 1 tasks_processing.go:71] worker 16 working on feature_gates task. W0428 15:37:41.320327 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0428 15:37:41.320338 1 tasks_processing.go:71] worker 12 working on sap_pods task. I0428 15:37:41.319889 1 tasks_processing.go:69] worker 58 listening for tasks. I0428 15:37:41.319897 1 tasks_processing.go:69] worker 13 listening for tasks. I0428 15:37:41.319899 1 tasks_processing.go:69] worker 51 listening for tasks. I0428 15:37:41.319902 1 tasks_processing.go:69] worker 37 listening for tasks. I0428 15:37:41.319907 1 tasks_processing.go:69] worker 52 listening for tasks. I0428 15:37:41.319902 1 tasks_processing.go:69] worker 18 listening for tasks. I0428 15:37:41.319905 1 tasks_processing.go:69] worker 14 listening for tasks. I0428 15:37:41.319912 1 tasks_processing.go:69] worker 38 listening for tasks. I0428 15:37:41.319914 1 tasks_processing.go:69] worker 53 listening for tasks. I0428 15:37:41.319920 1 tasks_processing.go:69] worker 39 listening for tasks. I0428 15:37:41.319918 1 tasks_processing.go:71] worker 9 working on silenced_alerts task. W0428 15:37:41.320433 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0428 15:37:41.320444 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 37.332µs to process 0 records I0428 15:37:41.320510 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 65.32µs to process 0 records I0428 15:37:41.319895 1 tasks_processing.go:69] worker 36 listening for tasks. I0428 15:37:41.319924 1 tasks_processing.go:71] worker 19 working on nodenetworkconfigurationpolicies task. I0428 15:37:41.320559 1 tasks_processing.go:71] worker 36 working on config_maps task. I0428 15:37:41.320564 1 tasks_processing.go:71] worker 18 working on container_runtime_configs task. I0428 15:37:41.320607 1 tasks_processing.go:71] worker 53 working on machine_autoscalers task. I0428 15:37:41.320611 1 tasks_processing.go:71] worker 39 working on metrics task. W0428 15:37:41.320634 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0428 15:37:41.320682 1 tasks_processing.go:71] worker 39 working on overlapping_namespace_uids task. I0428 15:37:41.320694 1 tasks_processing.go:71] worker 13 working on nodes task. I0428 15:37:41.320725 1 tasks_processing.go:71] worker 37 working on oauths task. I0428 15:37:41.320769 1 tasks_processing.go:71] worker 14 working on tsdb_status task. W0428 15:37:41.320794 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0428 15:37:41.320802 1 tasks_processing.go:71] worker 38 working on operators task. I0428 15:37:41.319928 1 tasks_processing.go:71] worker 20 working on image_registries task. I0428 15:37:41.320890 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 60.642µs to process 0 records I0428 15:37:41.320938 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 19.464µs to process 0 records I0428 15:37:41.320941 1 tasks_processing.go:71] worker 52 working on machines task. I0428 15:37:41.320953 1 tasks_processing.go:71] worker 14 working on clusterroles task. I0428 15:37:41.321031 1 tasks_processing.go:71] worker 9 working on support_secret task. I0428 15:37:41.320683 1 tasks_processing.go:71] worker 51 working on infrastructures task. I0428 15:37:41.319929 1 tasks_processing.go:69] worker 55 listening for tasks. I0428 15:37:41.321122 1 tasks_processing.go:71] worker 55 working on mutating_webhook_configurations task. I0428 15:37:41.319930 1 tasks_processing.go:71] worker 5 working on storage_cluster task. I0428 15:37:41.319933 1 tasks_processing.go:71] worker 21 working on ingress task. I0428 15:37:41.319934 1 tasks_processing.go:69] worker 62 listening for tasks. I0428 15:37:41.321442 1 tasks_processing.go:71] worker 62 working on sap_datahubs task. I0428 15:37:41.319935 1 tasks_processing.go:69] worker 41 listening for tasks. I0428 15:37:41.319935 1 tasks_processing.go:71] worker 6 working on ceph_cluster task. I0428 15:37:41.321494 1 tasks_processing.go:71] worker 41 working on aggregated_monitoring_cr_names task. I0428 15:37:41.319937 1 tasks_processing.go:69] worker 56 listening for tasks. I0428 15:37:41.321525 1 tasks_processing.go:71] worker 56 working on openstack_version task. I0428 15:37:41.319940 1 tasks_processing.go:71] worker 22 working on image task. I0428 15:37:41.319940 1 tasks_processing.go:69] worker 42 listening for tasks. I0428 15:37:41.321700 1 tasks_processing.go:71] worker 42 working on node_logs task. I0428 15:37:41.319942 1 tasks_processing.go:71] worker 7 working on openstack_controlplanes task. I0428 15:37:41.319942 1 tasks_processing.go:69] worker 59 listening for tasks. I0428 15:37:41.321788 1 tasks_processing.go:71] worker 59 working on version task. I0428 15:37:41.319944 1 tasks_processing.go:69] worker 43 listening for tasks. I0428 15:37:41.319945 1 tasks_processing.go:71] worker 23 working on install_plans task. I0428 15:37:41.321858 1 tasks_processing.go:71] worker 43 working on networks task. I0428 15:37:41.319947 1 tasks_processing.go:71] worker 8 working on cluster_apiserver task. I0428 15:37:41.319949 1 tasks_processing.go:69] worker 44 listening for tasks. I0428 15:37:41.322045 1 tasks_processing.go:71] worker 44 working on jaegers task. I0428 15:37:41.319950 1 tasks_processing.go:69] worker 60 listening for tasks. I0428 15:37:41.319946 1 tasks_processing.go:69] worker 57 listening for tasks. I0428 15:37:41.319954 1 tasks_processing.go:69] worker 45 listening for tasks. I0428 15:37:41.319950 1 tasks_processing.go:71] worker 24 working on proxies task. I0428 15:37:41.322104 1 tasks_processing.go:71] worker 45 working on monitoring_persistent_volumes task. I0428 15:37:41.322130 1 tasks_processing.go:71] worker 57 working on machine_sets task. I0428 15:37:41.319959 1 tasks_processing.go:69] worker 46 listening for tasks. I0428 15:37:41.322206 1 tasks_processing.go:71] worker 46 working on container_images task. I0428 15:37:41.319964 1 tasks_processing.go:69] worker 47 listening for tasks. I0428 15:37:41.319958 1 tasks_processing.go:71] worker 4 working on node_features task. I0428 15:37:41.322298 1 tasks_processing.go:71] worker 60 working on authentication task. I0428 15:37:41.322359 1 tasks_processing.go:71] worker 47 working on lokistack task. I0428 15:37:41.319958 1 tasks_processing.go:69] worker 61 listening for tasks. I0428 15:37:41.319962 1 tasks_processing.go:69] worker 63 listening for tasks. I0428 15:37:41.319968 1 tasks_processing.go:69] worker 48 listening for tasks. I0428 15:37:41.319972 1 tasks_processing.go:69] worker 29 listening for tasks. I0428 15:37:41.319973 1 tasks_processing.go:69] worker 27 listening for tasks. I0428 15:37:41.319973 1 tasks_processing.go:71] worker 1 working on openstack_dataplanedeployments task. I0428 15:37:41.319974 1 tasks_processing.go:69] worker 49 listening for tasks. I0428 15:37:41.319980 1 tasks_processing.go:69] worker 28 listening for tasks. I0428 15:37:41.319980 1 tasks_processing.go:69] worker 26 listening for tasks. I0428 15:37:41.319985 1 tasks_processing.go:69] worker 31 listening for tasks. I0428 15:37:41.319986 1 tasks_processing.go:69] worker 30 listening for tasks. I0428 15:37:41.319994 1 tasks_processing.go:69] worker 32 listening for tasks. I0428 15:37:41.319994 1 tasks_processing.go:69] worker 33 listening for tasks. I0428 15:37:41.319889 1 tasks_processing.go:69] worker 35 listening for tasks. I0428 15:37:41.320169 1 tasks_processing.go:71] worker 50 working on certificate_signing_requests task. I0428 15:37:41.320179 1 tasks_processing.go:71] worker 10 working on schedulers task. I0428 15:37:41.319922 1 tasks_processing.go:69] worker 54 listening for tasks. I0428 15:37:41.319917 1 tasks_processing.go:71] worker 3 working on machine_healthchecks task. I0428 15:37:41.320689 1 tasks_processing.go:71] worker 58 working on openshift_machine_api_events task. I0428 15:37:41.322447 1 tasks_processing.go:71] worker 49 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0428 15:37:41.319927 1 tasks_processing.go:69] worker 40 listening for tasks. I0428 15:37:41.322762 1 tasks_processing.go:71] worker 61 working on qemu_kubevirt_launcher_logs task. I0428 15:37:41.322767 1 tasks_processing.go:74] worker 40 stopped. I0428 15:37:41.322768 1 tasks_processing.go:71] worker 28 working on service_accounts task. I0428 15:37:41.322772 1 tasks_processing.go:71] worker 26 working on pod_network_connectivity_checks task. I0428 15:37:41.322874 1 tasks_processing.go:71] worker 30 working on crds task. I0428 15:37:41.322751 1 tasks_processing.go:74] worker 63 stopped. I0428 15:37:41.322959 1 tasks_processing.go:71] worker 32 working on olm_operators task. I0428 15:37:41.323022 1 tasks_processing.go:71] worker 31 working on dvo_metrics task. I0428 15:37:41.323052 1 tasks_processing.go:71] worker 27 working on nodenetworkstates task. I0428 15:37:41.322757 1 tasks_processing.go:71] worker 33 working on cost_management_metrics_configs task. I0428 15:37:41.323080 1 tasks_processing.go:71] worker 29 working on machine_configs task. I0428 15:37:41.323134 1 tasks_processing.go:71] worker 54 working on openshift_logging task. I0428 15:37:41.323233 1 tasks_processing.go:71] worker 35 working on pdbs task. I0428 15:37:41.322762 1 tasks_processing.go:71] worker 48 working on openstack_dataplanenodesets task. I0428 15:37:41.324649 1 tasks_processing.go:74] worker 0 stopped. I0428 15:37:41.324668 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 4.848121ms to process 0 records I0428 15:37:41.325809 1 tasks_processing.go:74] worker 2 stopped. I0428 15:37:41.325827 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 6.000933ms to process 0 records I0428 15:37:41.326858 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0428 15:37:41.326943 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0428 15:37:41.326950 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0428 15:37:41.328204 1 tasks_processing.go:74] worker 19 stopped. I0428 15:37:41.328220 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 7.642684ms to process 0 records I0428 15:37:41.330712 1 controller.go:129] Initializing last reported time to 0001-01-01T00:00:00Z I0428 15:37:41.330735 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0428 15:37:41.330739 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0428 15:37:41.330744 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0428 15:37:41.330757 1 controller.go:531] The operator is still being initialized I0428 15:37:41.330762 1 controller.go:554] The operator is healthy I0428 15:37:41.331185 1 tasks_processing.go:74] worker 16 stopped. I0428 15:37:41.331345 1 recorder.go:75] Recording config/featuregate with fingerprint=b9f1a05e7d10d0485389baedfedee3f64c1e00b9523b8d5fbe3e93f0b71aca0e I0428 15:37:41.331369 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 10.840368ms to process 1 records I0428 15:37:41.339104 1 tasks_processing.go:74] worker 53 stopped. I0428 15:37:41.339121 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 18.468461ms to process 0 records I0428 15:37:41.339135 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 18.548039ms to process 0 records I0428 15:37:41.339144 1 tasks_processing.go:74] worker 18 stopped. I0428 15:37:41.339187 1 tasks_processing.go:74] worker 12 stopped. I0428 15:37:41.339194 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 18.837643ms to process 0 records I0428 15:37:41.339297 1 tasks_processing.go:74] worker 44 stopped. I0428 15:37:41.339311 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 17.239408ms to process 0 records E0428 15:37:41.339439 1 gather_node_features.go:86] GatherNodeFeatures: NodeFeatures resource not found in openshift-nfd namespace (may not be installed) I0428 15:37:41.339442 1 tasks_processing.go:74] worker 57 stopped. I0428 15:37:41.339461 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 17.292943ms to process 0 records I0428 15:37:41.339479 1 gather.go:177] gatherer "clusterconfig" function "node_features" took 17.224883ms to process 0 records I0428 15:37:41.339484 1 tasks_processing.go:74] worker 4 stopped. I0428 15:37:41.339579 1 tasks_processing.go:74] worker 47 stopped. I0428 15:37:41.339594 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 17.109632ms to process 0 records I0428 15:37:41.340750 1 tasks_processing.go:74] worker 5 stopped. I0428 15:37:41.340759 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 19.612418ms to process 0 records I0428 15:37:41.342426 1 tasks_processing.go:74] worker 11 stopped. I0428 15:37:41.343902 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=58a6976fd7d0d4f05429bdb867ab49132c9f727d1370ea6cd7080ba64de89c4f I0428 15:37:41.343925 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 22.301666ms to process 1 records E0428 15:37:41.343943 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0428 15:37:41.343951 1 gather.go:177] gatherer "clusterconfig" function "machines" took 22.701467ms to process 0 records I0428 15:37:41.343959 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 22.212574ms to process 0 records I0428 15:37:41.343959 1 tasks_processing.go:74] worker 52 stopped. I0428 15:37:41.343971 1 tasks_processing.go:74] worker 62 stopped. I0428 15:37:41.344028 1 recorder.go:75] Recording config/proxy with fingerprint=c9c804c5aa56aaa0697d3f985ba3eec116d176ff93bb08337294498b3ac75f52 I0428 15:37:41.344035 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 21.591897ms to process 1 records I0428 15:37:41.344040 1 tasks_processing.go:74] worker 24 stopped. I0428 15:37:41.344161 1 tasks_processing.go:74] worker 34 stopped. I0428 15:37:41.344432 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=9686bfb6c26ce1cafca59c12f4d63691cc0879b898ad47106bb7fb9ec1894a2d I0428 15:37:41.344602 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=ba1a0a9bd014b259d009115e8fcc67b753e627e5dc87f0205f21df8aa481c751 I0428 15:37:41.344655 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=f7d18becdec20033bfee6f453aaa9ac08966faa1a153c856aed4313b81755e23 I0428 15:37:41.344712 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=c68f6cd2b9cc9fd5e7ab49a50fd010f22023f8dbb46cfa86c941d2af394dbd37 I0428 15:37:41.344761 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=64864463c78221dabf3425b8c222123e09198b7c2fb3f4bf038cecfc8edd8873 I0428 15:37:41.344815 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=cac9d045c144b32f561773d6d2ac493af3d0175abc81133bfe99a702502e4167 I0428 15:37:41.344867 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=d2856c5c97018e5c30b86268da63e07915dc7594945a29c56ade98bbd0c682ac I0428 15:37:41.344924 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=f3156a958002991a1d1d7720b6268d91de1ad3b7eef8aa86d674eaa4e469f633 I0428 15:37:41.344949 1 base_controller.go:82] Caches are synced for ConfigController I0428 15:37:41.344961 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0428 15:37:41.344962 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=aef0066dfb3fe1a947fbeac37e45ae719ba733319320cb031371511c4da16632 I0428 15:37:41.345039 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=952aad632be5a527ca47aa61940dfeafa20cf9c42fe36c130b4f5a5cb9c61c6c I0428 15:37:41.345107 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=d704161d61f4ed47738e605f37585415b4aa32a374e02b1d758fec9211ba1198 I0428 15:37:41.345120 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 24.018521ms to process 11 records I0428 15:37:41.345211 1 tasks_processing.go:74] worker 60 stopped. I0428 15:37:41.345407 1 recorder.go:75] Recording config/authentication with fingerprint=485a1e63a7be9d1cd5cf5c34bf25b3a34d9c47dcdbfba2a58485a7bb10f5cd59 I0428 15:37:41.345422 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 22.622114ms to process 1 records I0428 15:37:41.346782 1 tasks_processing.go:74] worker 21 stopped. I0428 15:37:41.346964 1 recorder.go:75] Recording config/ingress with fingerprint=6213581b4bc69b0247b7b64c9459677fa3ed8a547af9375d35eed38fa122a426 I0428 15:37:41.346980 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 25.455162ms to process 1 records I0428 15:37:41.347028 1 tasks_processing.go:74] worker 9 stopped. E0428 15:37:41.347038 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0428 15:37:41.347045 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 25.97484ms to process 0 records I0428 15:37:41.347557 1 tasks_processing.go:74] worker 6 stopped. I0428 15:37:41.347575 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 26.070884ms to process 0 records I0428 15:37:41.348917 1 tasks_processing.go:74] worker 1 stopped. I0428 15:37:41.348930 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 26.346138ms to process 0 records I0428 15:37:41.349015 1 tasks_processing.go:74] worker 13 stopped. I0428 15:37:41.349512 1 recorder.go:75] Recording config/node/ip-10-0-0-9.ec2.internal with fingerprint=cc8707b5727067d34f3855624277bae1b9e6369af59213190e3aab0a18730170 I0428 15:37:41.349612 1 recorder.go:75] Recording config/node/ip-10-0-1-106.ec2.internal with fingerprint=453eba88a77087486be2be129a6a0d4c329c0acf914b066aaec869127496764c I0428 15:37:41.349695 1 recorder.go:75] Recording config/node/ip-10-0-2-7.ec2.internal with fingerprint=14df0f4dfbe2120b7ad4dbf5a0f913f069bcdbe22d300be88016522ed3182443 I0428 15:37:41.349708 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 28.304499ms to process 3 records I0428 15:37:41.349794 1 tasks_processing.go:74] worker 37 stopped. I0428 15:37:41.349992 1 recorder.go:75] Recording config/oauth with fingerprint=74fe96ff3f97999ba27e32fe584fa4a49c9c10d90a85a0311ef557d847898c90 I0428 15:37:41.350006 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 28.447295ms to process 1 records I0428 15:37:41.350101 1 tasks_processing.go:74] worker 8 stopped. I0428 15:37:41.350200 1 recorder.go:75] Recording config/apiserver with fingerprint=5deda927964bbff6f949fc94e9fc22bd10a1f13c3b5fb0842e613b025184a496 I0428 15:37:41.350212 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 27.557708ms to process 1 records I0428 15:37:41.350295 1 tasks_processing.go:74] worker 20 stopped. I0428 15:37:41.350662 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=831186954aa6a110cd1c7a9f91ed25104c1644d820d9a8f3acff0283f4033e3d I0428 15:37:41.350676 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 29.181458ms to process 1 records I0428 15:37:41.351987 1 tasks_processing.go:74] worker 7 stopped. I0428 15:37:41.352002 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 30.270456ms to process 0 records I0428 15:37:41.352220 1 tasks_processing.go:74] worker 27 stopped. I0428 15:37:41.352237 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 29.15146ms to process 0 records I0428 15:37:41.352247 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 29.07648ms to process 0 records I0428 15:37:41.352254 1 tasks_processing.go:74] worker 54 stopped. I0428 15:37:41.352552 1 tasks_processing.go:74] worker 33 stopped. I0428 15:37:41.352563 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 29.479439ms to process 0 records I0428 15:37:41.352774 1 tasks_processing.go:74] worker 48 stopped. I0428 15:37:41.352789 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 29.447397ms to process 0 records I0428 15:37:41.362629 1 tasks_processing.go:74] worker 15 stopped. I0428 15:37:41.362740 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=078867141628b3a6491fa147a4848cfff929e5f8d8d42186cb1604ec86e481aa I0428 15:37:41.362757 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=faf1694dc274ed2ba48e54ba8536fa1ddff5daa6c1133fdb2d2021fd7919262e I0428 15:37:41.362763 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 42.410978ms to process 2 records I0428 15:37:41.363380 1 tasks_processing.go:74] worker 56 stopped. I0428 15:37:41.363389 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 41.843339ms to process 0 records I0428 15:37:41.363585 1 tasks_processing.go:74] worker 26 stopped. E0428 15:37:41.363594 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0428 15:37:41.363602 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 40.793849ms to process 0 records I0428 15:37:41.363709 1 tasks_processing.go:74] worker 51 stopped. I0428 15:37:41.364746 1 recorder.go:75] Recording config/infrastructure with fingerprint=022baa490c2378735026c2cc49d7e16b1e3b4c8452a6102026cdadc7bf2bd819 I0428 15:37:41.364767 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 42.618012ms to process 1 records I0428 15:37:41.364783 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 42.916708ms to process 0 records I0428 15:37:41.364791 1 tasks_processing.go:74] worker 42 stopped. I0428 15:37:41.366035 1 tasks_processing.go:74] worker 43 stopped. I0428 15:37:41.366188 1 recorder.go:75] Recording config/network with fingerprint=183befc375aec3b3fe032dce7775a084cb26c9f4e05d586e1a7731387b02ec15 I0428 15:37:41.366204 1 gather.go:177] gatherer "clusterconfig" function "networks" took 44.161672ms to process 1 records I0428 15:37:41.366269 1 recorder.go:75] Recording config/image with fingerprint=1f4e542eee4370e84b38bf22e792332aaf0aff5afbf1d5ec06a079e95ba7670e I0428 15:37:41.366276 1 gather.go:177] gatherer "clusterconfig" function "image" took 44.497373ms to process 1 records I0428 15:37:41.366282 1 tasks_processing.go:74] worker 22 stopped. I0428 15:37:41.366315 1 tasks_processing.go:74] worker 10 stopped. I0428 15:37:41.366375 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=dea3963d6b6852e8e73818b1e2e876b9ab2929001b2c5b2ff6d10d0be01ecf29 I0428 15:37:41.366387 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 43.829029ms to process 1 records I0428 15:37:41.368060 1 tasks_processing.go:74] worker 45 stopped. I0428 15:37:41.368077 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 45.943458ms to process 0 records I0428 15:37:41.370220 1 tasks_processing.go:74] worker 3 stopped. E0428 15:37:41.370345 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0428 15:37:41.370386 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 47.718487ms to process 0 records I0428 15:37:41.372792 1 tasks_processing.go:74] worker 32 stopped. I0428 15:37:41.372841 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 49.821969ms to process 0 records I0428 15:37:41.372940 1 tasks_processing.go:74] worker 55 stopped. I0428 15:37:41.373378 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=da84560f131ddeaa5023179330dfc5b1ab126539c6a65514266b501666ca315a I0428 15:37:41.373461 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=b4e770c4a6556c18deac0488cf88abbac5ab2038cdf18d2ba3613452676fdd41 I0428 15:37:41.373506 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=d179a47796ab49fd45e337ffb0b5859c7b49d2bacf5d39aadc5e2fc09263cedc I0428 15:37:41.373521 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 51.778945ms to process 3 records I0428 15:37:41.373536 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 50.949922ms to process 0 records I0428 15:37:41.373549 1 tasks_processing.go:74] worker 58 stopped. I0428 15:37:41.373550 1 tasks_processing.go:74] worker 29 stopped. E0428 15:37:41.373566 1 gather.go:140] gatherer "clusterconfig" function "machine_configs" failed with the error: getting MachineConfigPools failed: the server could not find the requested resource (get machineconfigpools.machineconfiguration.openshift.io) I0428 15:37:41.373640 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0428 15:37:41.373649 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 50.427768ms to process 1 records I0428 15:37:41.374332 1 tasks_processing.go:74] worker 39 stopped. I0428 15:37:41.374368 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0428 15:37:41.374384 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 53.624292ms to process 1 records I0428 15:37:41.375328 1 tasks_processing.go:74] worker 50 stopped. I0428 15:37:41.375342 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 52.702106ms to process 0 records I0428 15:37:41.377960 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0428 15:37:41.377979 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0428 15:37:41.378148 1 operator.go:328] started I0428 15:37:41.378168 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0428 15:37:41.378321 1 tasks_processing.go:74] worker 35 stopped. I0428 15:37:41.378443 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=3b1e04b7825804d0cdb5b6465fbbddc705958daf8c93686378b37256cd53978f I0428 15:37:41.378470 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=e8c2340ce40611132a42c0a0dee77a673325e00baa6c44e4a88178d6be07b32e I0428 15:37:41.378486 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=2f2a2ca126659a4706e9be0f9d76f49ca736a6dac803fb4b1e14e141cc6133e9 I0428 15:37:41.378493 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 55.065034ms to process 3 records W0428 15:37:41.383008 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0428 15:37:41.383012 1 tasks_processing.go:74] worker 14 stopped. I0428 15:37:41.383226 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=d5e19a17543080ba97f217799c7ed47a83d0adb4eafb06119e738a7f4b13ac81 I0428 15:37:41.383333 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=8ac702c63034a7cef8806362f82155a31e4db39ddb234cad87e490a9c61de7ef I0428 15:37:41.383347 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 62.045786ms to process 2 records I0428 15:37:41.388500 1 configmapobserver.go:84] configmaps "insights-config" not found I0428 15:37:41.389302 1 tasks_processing.go:74] worker 30 stopped. I0428 15:37:41.390075 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=0ee6296f077ce649156892ff984f4e2a3491574f79d5cc301d7679806095ecd7 I0428 15:37:41.390469 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=ec9771adfedbbf02bb9a110a5467623bb0666ddec8d5cd1b909eb8a7da594f42 I0428 15:37:41.390487 1 gather.go:177] gatherer "clusterconfig" function "crds" took 66.406771ms to process 2 records I0428 15:37:41.392274 1 controller.go:254] Source scaController *sca.Controller is not ready I0428 15:37:41.392286 1 controller.go:254] Source clusterTransferController *clustertransfer.Controller is not ready I0428 15:37:41.392290 1 controller.go:254] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0428 15:37:41.392293 1 controller.go:254] Source periodic-conditional *controllerstatus.Simple is not ready I0428 15:37:41.392295 1 controller.go:254] Source periodic-workloads *controllerstatus.Simple is not ready I0428 15:37:41.392312 1 controller.go:531] The operator is still being initialized I0428 15:37:41.392318 1 controller.go:554] The operator is healthy I0428 15:37:41.403971 1 prometheus_rules.go:88] Prometheus rules successfully created I0428 15:37:41.404757 1 tasks_processing.go:74] worker 41 stopped. I0428 15:37:41.404772 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 83.240395ms to process 0 records E0428 15:37:41.405988 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27ed0d5c36-8ad1-47f6-adc1-ae627bd95e26%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.9:60631->172.30.0.10:53: read: connection refused I0428 15:37:41.406001 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27ed0d5c36-8ad1-47f6-adc1-ae627bd95e26%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.9:60631->172.30.0.10:53: read: connection refused I0428 15:37:41.410819 1 gather_logs.go:145] no pods in namespace were found I0428 15:37:41.410834 1 tasks_processing.go:74] worker 61 stopped. I0428 15:37:41.410844 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 88.061419ms to process 0 records I0428 15:37:41.414057 1 tasks_processing.go:74] worker 59 stopped. I0428 15:37:41.414352 1 recorder.go:75] Recording config/version with fingerprint=69886ac66f026ab863cb5ebdadfdfd5b0d150c3f7f17a0e97331d88b8b2ae22c I0428 15:37:41.414368 1 recorder.go:75] Recording config/id with fingerprint=7910ec7bdce3336261b6f845b7d40652422f4a98eda661d09bb51ee833216acf I0428 15:37:41.414378 1 gather.go:177] gatherer "clusterconfig" function "version" took 92.250589ms to process 2 records I0428 15:37:41.435652 1 tasks_processing.go:74] worker 49 stopped. I0428 15:37:41.435673 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 113.186476ms to process 0 records I0428 15:37:41.437536 1 tasks_processing.go:74] worker 46 stopped. I0428 15:37:41.437643 1 recorder.go:75] Recording config/running_containers with fingerprint=33d6c661e6ede7959b578ac768384a7f5d7ef7520979846dc22668c0847517a9 I0428 15:37:41.437664 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 115.313966ms to process 1 records I0428 15:37:41.440399 1 tasks_processing.go:74] worker 36 stopped. E0428 15:37:41.440421 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0428 15:37:41.440434 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0428 15:37:41.440439 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0428 15:37:41.440450 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=ad286723f58bdcfc37aeba3ec5b4110c08e1af59cd34d14b4bfaab02d18e4856 I0428 15:37:41.440498 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0428 15:37:41.440507 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0428 15:37:41.440512 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=2ab5ab7b1b10d7fcf1197bb24dea7c90f400e4effc18c7356873209d54fdf84b I0428 15:37:41.440516 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0428 15:37:41.440554 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0428 15:37:41.440564 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0428 15:37:41.440573 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 119.820004ms to process 7 records I0428 15:37:41.458266 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0428 15:37:41.462069 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.9:39829->172.30.0.10:53: read: connection refused I0428 15:37:41.462106 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.9:39829->172.30.0.10:53: read: connection refused I0428 15:37:41.478538 1 base_controller.go:82] Caches are synced for LoggingSyncer I0428 15:37:41.478578 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0428 15:37:41.495430 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0428 15:37:41.501350 1 tasks_processing.go:74] worker 17 stopped. E0428 15:37:41.501366 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0428 15:37:41.501372 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2pv84nia58gje8norbmmjqtj26oprnd8-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2pv84nia58gje8norbmmjqtj26oprnd8-primary-cert-bundle-secret" not found I0428 15:37:41.501434 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=f7f5751398ace9511543201acf78ea359b690d7c53aaccf0028e67a3d8ae1c7a I0428 15:37:41.501449 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 181.227113ms to process 1 records I0428 15:37:41.509877 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0428 15:37:41.936215 1 gather_cluster_operator_pods_and_events.go:121] Found 14 pods with 15 containers I0428 15:37:41.936229 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 1677721 bytes I0428 15:37:41.938201 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-r892r pod in namespace openshift-dns (previous: false). I0428 15:37:42.034855 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-r892r pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-r892r\" is waiting to start: ContainerCreating" I0428 15:37:42.034877 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-r892r\" is waiting to start: ContainerCreating" I0428 15:37:42.034889 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-r892r pod in namespace openshift-dns (previous: false). I0428 15:37:42.186039 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-r892r pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-r892r\" is waiting to start: ContainerCreating" I0428 15:37:42.186061 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-r892r\" is waiting to start: ContainerCreating" I0428 15:37:42.186074 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-h8stv pod in namespace openshift-dns (previous: false). W0428 15:37:42.382732 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0428 15:37:42.414584 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0428 15:37:42.414602 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-qv85l pod in namespace openshift-dns (previous: false). I0428 15:37:42.624368 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0428 15:37:42.624397 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-t5fvn pod in namespace openshift-dns (previous: false). I0428 15:37:42.806761 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0428 15:37:42.806780 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-6669988f88-tj54w pod in namespace openshift-image-registry (previous: false). I0428 15:37:42.807424 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0428 15:37:42.975752 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0428 15:37:42.975772 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7fc7c49d95-ctrtf pod in namespace openshift-image-registry (previous: false). I0428 15:37:43.173218 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0428 15:37:43.173237 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7fc7c49d95-dgxkp pod in namespace openshift-image-registry (previous: false). I0428 15:37:43.375382 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-7fc7c49d95-dgxkp pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-7fc7c49d95-dgxkp\" is waiting to start: ContainerCreating" I0428 15:37:43.375399 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-7fc7c49d95-dgxkp\" is waiting to start: ContainerCreating" I0428 15:37:43.375412 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-8cb28 pod in namespace openshift-image-registry (previous: false). W0428 15:37:43.384345 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0428 15:37:43.406696 1 tasks_processing.go:74] worker 38 stopped. I0428 15:37:43.406752 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=1c0eb10b5e3d9a85318cdcf33942ed4b3631dfa5e789a5323c5832220f0e0d59 I0428 15:37:43.406801 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=c0dec3aeea945ba0b00d7410703a9520d014bdfce29f8fc852ecd6fac8603885 I0428 15:37:43.406834 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0428 15:37:43.406858 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=9f0cd4321d887ca21590acbf331d90a4d06f9228caf9d6422a72b3ac67f2a754 I0428 15:37:43.406875 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0428 15:37:43.406898 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=e00f8f2cd507763ba565486752475b1b3cf2d12bf9e7e848bedadcc12f112bcb I0428 15:37:43.406948 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=409bdad2a9f0276fa36444cddbaa5ecf84280c72cee70bfbabea14c0f1f56f0b I0428 15:37:43.406973 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=14eba7e257c53119afba7bff6c16c3edcfb5baf12e63737bd87c1eee6fd50988 I0428 15:37:43.406987 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=6825b646c4ba3774e3dbe4fa30962ce63fe03d382d92451a69381d771ee5fde4 I0428 15:37:43.407004 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=d31999bcf22bf8f86232c1ceee2548cd43a2097bb7ff12bb5e3b97caa3c041cc I0428 15:37:43.407014 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0428 15:37:43.407030 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=5428d003f7f6ed9948f1a9c160ff6f9484b23888b5328aa8d28d7267b4ed4505 I0428 15:37:43.407040 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0428 15:37:43.407057 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=280516d3001819f8cf2ba4b35cc7e0d8d12cd0f042ee20621a657b5da7e2c324 I0428 15:37:43.407066 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0428 15:37:43.407080 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=151dd453e54bcef55c4bbfd800d04754cce7dba7c73b4560aed755d863d02864 I0428 15:37:43.407106 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0428 15:37:43.407125 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=24601bcc6aa1317dfad71a53d56cabbe6c2bb5c2558eef501b2b33c46c720831 I0428 15:37:43.407254 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=92760e6f90b9e63d9e92c3e0cfa5f6dfb688c974eb77f465c4f86211da7a637e I0428 15:37:43.407264 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0428 15:37:43.407272 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0428 15:37:43.407292 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0428 15:37:43.407315 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=b32f134a8ef0de7ab6bdda81c0dcd36536cd86e1c89bf2699ee3d54369751625 I0428 15:37:43.407340 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=524ed5091f0d43734b879ae2be6541f3d8f571ebf6fc91ad9ec1fe068f001362 I0428 15:37:43.407349 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0428 15:37:43.407365 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=56c847358e235f4116341fd370b6137f59b4948e4e2355ad55d6ed5a18acfeb4 I0428 15:37:43.407375 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0428 15:37:43.407414 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=71ce46c46e6363eec408123b6704c49d34abfa21850c3677acfa965a70b802ce I0428 15:37:43.407431 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=88b989f2e866a482976ccf7b347f1fb0e0c857cd6b496afdbefbe137c5a995aa I0428 15:37:43.407447 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=a6d898195a671944011fe849f86604f33dcaf22c79db76b10bffcaf528c2aa47 I0428 15:37:43.407463 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=cd48246332e851f3b865b48b42e76a42067e42bfc369fa8094b89e06c772f737 I0428 15:37:43.407486 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=10456c97a14787701b00e7d1ae050512bb61ceab60f95a1a5de976c92ea762cd I0428 15:37:43.407497 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0428 15:37:43.407524 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=a89d4381340cc49c538df86e590f0cff766ca4bef664ee2fc426e5b6353f1365 I0428 15:37:43.407540 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0428 15:37:43.407549 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0428 15:37:43.407556 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.085875732s to process 36 records I0428 15:37:43.582606 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0428 15:37:43.582623 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-kw46n pod in namespace openshift-image-registry (previous: false). I0428 15:37:43.775319 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0428 15:37:43.775337 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-wxm7m pod in namespace openshift-image-registry (previous: false). I0428 15:37:43.983194 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0428 15:37:43.983216 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5f578f8577-xnnpr pod in namespace openshift-ingress (previous: false). I0428 15:37:44.174922 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5f578f8577-xnnpr pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5f578f8577-xnnpr\" is waiting to start: ContainerCreating" I0428 15:37:44.174941 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5f578f8577-xnnpr\" is waiting to start: ContainerCreating" I0428 15:37:44.174954 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6d4c95c6c5-bjdcq pod in namespace openshift-ingress (previous: false). I0428 15:37:44.374504 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0428 15:37:44.374522 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-6d4c95c6c5-nsvcp pod in namespace openshift-ingress (previous: false). W0428 15:37:44.383536 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0428 15:37:44.575000 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-6d4c95c6c5-nsvcp pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-6d4c95c6c5-nsvcp\" is waiting to start: ContainerCreating" I0428 15:37:44.575018 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-6d4c95c6c5-nsvcp\" is waiting to start: ContainerCreating" I0428 15:37:44.575029 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-qnplk pod in namespace openshift-ingress-canary (previous: false). I0428 15:37:44.775409 1 tasks_processing.go:74] worker 25 stopped. I0428 15:37:44.775496 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=9694b9d77d016891aa77b15466b8530bcfe0291b47462b4b525dc1e52cc913ba I0428 15:37:44.775538 1 recorder.go:75] Recording events/openshift-dns with fingerprint=15cac3c4876adf793a32e358a67e2517d2ee4344280b0ba57e5322828dfce4eb I0428 15:37:44.775621 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=c20f1b6b622b7a83b6fe309e1c568d8118931a0e5fb0a2ec7ccdd9db9109d061 I0428 15:37:44.775651 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=e2328804343bffbe43371961c1695284a4e0fc41eb8ff3f1d80cd64c952b848b I0428 15:37:44.775697 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=5a3eb3017ba928556d0618e6bfc8d01a23d9cefe7b5f2ec97b2c672296ee3127 I0428 15:37:44.775718 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=fd1d372312ecd08b0a8782c914ff4d48174934c94135a1bed4328983462b24d7 I0428 15:37:44.775725 1 recorder.go:75] Recording config/pod/openshift-ingress-canary/logs/ingress-canary-qnplk/serve-healthcheck-canary_current.log with fingerprint=7549d72ec55860cb7b321fb91b334f3cf7c0bae294d968c76aef8be54eceba7a I0428 15:37:44.775733 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 3.455378214s to process 7 records W0428 15:37:45.383976 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0428 15:37:46.383294 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0428 15:37:46.383321 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0428 15:37:46.383337 1 tasks_processing.go:74] worker 31 stopped. E0428 15:37:46.383349 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0428 15:37:46.383360 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0428 15:37:46.383374 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0428 15:37:46.383386 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.06029914s to process 1 records I0428 15:37:50.339347 1 configmapobserver.go:84] configmaps "insights-config" not found I0428 15:37:50.409716 1 configmapobserver.go:84] configmaps "insights-config" not found I0428 15:37:50.608011 1 configmapobserver.go:84] configmaps "insights-config" not found I0428 15:37:53.981009 1 tasks_processing.go:74] worker 23 stopped. I0428 15:37:53.981056 1 recorder.go:75] Recording config/installplans with fingerprint=7b887df561a3a9e6ef0dc672845aa5d56e348505006b7496d3a2f83892b0c95b I0428 15:37:53.981071 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.659143347s to process 1 records I0428 15:37:54.729669 1 tasks_processing.go:74] worker 28 stopped. I0428 15:37:54.729990 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=efca7636f6879171afb7da6af2a0d8b2c35c2233fe3c0388ddda9fdab3080fb6 I0428 15:37:54.730014 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.406877731s to process 1 records E0428 15:37:54.730128 1 periodic.go:254] "Unhandled Error" err="clusterconfig failed after 13.41s with: function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machine_configs\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0428 15:37:54.731241 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "machines" failed with an error, function "support_secret" failed with an error, function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "machine_configs" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0428 15:37:54.731257 1 periodic.go:216] Running workloads gatherer I0428 15:37:54.731273 1 tasks_processing.go:45] number of workers: 2 I0428 15:37:54.731287 1 tasks_processing.go:69] worker 1 listening for tasks. I0428 15:37:54.731293 1 tasks_processing.go:71] worker 1 working on workload_info task. I0428 15:37:54.731359 1 tasks_processing.go:69] worker 0 listening for tasks. I0428 15:37:54.731453 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0428 15:37:54.753354 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 21s for image data I0428 15:37:54.761311 1 tasks_processing.go:74] worker 0 stopped. I0428 15:37:54.761328 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 29.833741ms to process 0 records I0428 15:37:54.790255 1 gather_workloads_info.go:387] No image sha256:730d1b6988025bef0daa3a9a5d8467ec4a26b0382cc52f91c3375b4590d3518a (38ms) I0428 15:37:54.799378 1 gather_workloads_info.go:387] No image sha256:a043239802b3eb8b323d285193d2527fad0ecec98ca91d188a3472a2fac8ae04 (9ms) I0428 15:37:54.809628 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (10ms) I0428 15:37:54.819154 1 gather_workloads_info.go:387] No image sha256:934d8e8c50f3c609b8eea80d1051111fe3d066fe8c65c79572072ae55fcb0a86 (10ms) I0428 15:37:54.828022 1 gather_workloads_info.go:387] No image sha256:5808401268394502d335281ea1a294b07210461b986b58f91d7d1f29c0029c6d (9ms) I0428 15:37:54.837882 1 gather_workloads_info.go:387] No image sha256:875c77e5d144f03fb91d8cee0259f6966683ca88d1bf818dbf4652c16b70312c (10ms) I0428 15:37:54.847155 1 gather_workloads_info.go:387] No image sha256:0a4dfb8d4c1b3849319d45b4c54dff26a7238a2c08fcaa121f93073e95ab12e8 (9ms) I0428 15:37:54.855816 1 gather_workloads_info.go:387] No image sha256:4733236617781e3469ffffb15e4daaa1f14ea8e1c52b426a3787a4f1f2945424 (9ms) I0428 15:37:54.864438 1 gather_workloads_info.go:387] No image sha256:4b748615a43be416d52c33df03e2cdec89ededded1d6a2ade167c5a955d56e5f (9ms) I0428 15:37:54.873386 1 gather_workloads_info.go:387] No image sha256:15677f0b70e6aa2dfaf088e45fc1a425c22bf6fda326b8116f87e88b6694dfab (9ms) I0428 15:37:54.882513 1 gather_workloads_info.go:387] No image sha256:30e597ec5d6bb96ff70a4f8688c748b659cd4fd5d73d222e8701821d236795c5 (9ms) I0428 15:37:54.963819 1 gather_workloads_info.go:387] No image sha256:000105ef5150e7079b90a613fb9e6193e2a6ef9b1908d2dce44f2395d4fd070f (81ms) I0428 15:37:55.062830 1 gather_workloads_info.go:387] No image sha256:48e883a3932aea9457f55cd4628d21397d429d4828f42f95d1c903d9d5395bde (99ms) I0428 15:37:55.165217 1 gather_workloads_info.go:387] No image sha256:ce138e8cf5b96557b1864ef6f27c2608bdca59be2611804366cef7169c36291e (102ms) I0428 15:37:55.262955 1 gather_workloads_info.go:387] No image sha256:6814f5fced0ef219d06374011c68a11a3da788a764a00a69ade435466d9ee240 (98ms) I0428 15:37:55.364043 1 gather_workloads_info.go:387] No image sha256:25c148fd380b1a9db3f6039d2e0eabc489a954921452391390ba9192b2325678 (101ms) I0428 15:37:55.463039 1 gather_workloads_info.go:387] No image sha256:ab60623bb32f7e75fca71ef65137731cae347a21c7a4091dfd583fa00732721c (99ms) I0428 15:37:55.567158 1 gather_workloads_info.go:387] No image sha256:5aaea0419169e55832cc27acfe0fe3b9513a343d6bdf71d3da1575ed322245d0 (104ms) I0428 15:37:55.663539 1 gather_workloads_info.go:387] No image sha256:5c6d21c3f97366bc7ab57031cc027b67405a684bd804ce364ed5998b0685eaca (96ms) I0428 15:37:55.762671 1 gather_workloads_info.go:387] No image sha256:55b1db6038c5beaed54c626e3343b7a8589cc0be8dc41d1a66b4deab766ff520 (99ms) I0428 15:37:55.862928 1 gather_workloads_info.go:387] No image sha256:89277d8d4560d71db88c2dcc67c992a24544ca21810920b609c1d49d53b4a287 (100ms) I0428 15:37:55.965359 1 gather_workloads_info.go:387] No image sha256:637d41f067a5239096fc22b135181cda5113da833f1370e7a73965e83792e93a (102ms) I0428 15:37:56.066867 1 gather_workloads_info.go:387] No image sha256:d64bea34bf3e1bb0b3a701c3ff14e66665afc1b050f28124ad7e6888eaec3a81 (101ms) I0428 15:37:56.163702 1 gather_workloads_info.go:387] No image sha256:673ebc8cc22c56c8f410e011b2fa950d28cf7b6420e17fdb6580d6cb10523384 (97ms) I0428 15:37:56.263685 1 gather_workloads_info.go:387] No image sha256:5ac9c549d65fc1d8bc900773bebee43e9192bcec1bb5fa46afb4597230c16ac7 (100ms) I0428 15:37:56.365342 1 gather_workloads_info.go:387] No image sha256:8f881f6f0498eaba860d711dc8e9dc066db8728f22abedb82e1dc9a8e1c30046 (102ms) I0428 15:37:56.463889 1 gather_workloads_info.go:387] No image sha256:5b6212b8f539f08e78417d8a4b7485ca0b4e7927cacd7b752742a28841bc8ccd (99ms) I0428 15:37:56.562916 1 gather_workloads_info.go:387] No image sha256:e3732e356ae2324565c74cae57d5d016917314fa293ddaa3a68ee9ae030c6f07 (99ms) I0428 15:37:56.663179 1 gather_workloads_info.go:387] No image sha256:084aa9b0f8a6d478549dc384d4e66da13ee9b25cc98531da861cc19dee2a9e8f (100ms) I0428 15:37:56.766165 1 gather_workloads_info.go:387] No image sha256:d6fbe0075cbb12bfd287c973704fadf97154c7f73e370733d976a40835e9155a (103ms) I0428 15:37:56.863539 1 gather_workloads_info.go:387] No image sha256:08c5a78c8a5af04c549e2273aaf4bb452a75bf038d68aa9d01bb2aff66c30e90 (97ms) I0428 15:37:56.964062 1 gather_workloads_info.go:387] No image sha256:3bba1358d4a0ae878ff491c0c2cbfffe60649e110b40342b878fe8fa332f8858 (101ms) I0428 15:37:57.063925 1 gather_workloads_info.go:387] No image sha256:e1ba458cf6f0b3606c90880da72db8ab99cd11040bae84baebe3ff2e0d1ea075 (100ms) I0428 15:37:57.063960 1 tasks_processing.go:74] worker 1 stopped. E0428 15:37:57.063972 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0428 15:37:57.064274 1 recorder.go:75] Recording config/workload_info with fingerprint=ebee1b87c9e8b24f19253386a996f55d04968b40cb0669107ec945ff0539a330 I0428 15:37:57.064292 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.332658347s to process 1 records E0428 15:37:57.064327 1 periodic.go:254] "Unhandled Error" err="workloads failed after 2.333s with: function \"workload_info\" failed with an error" I0428 15:37:57.065432 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0428 15:37:57.065444 1 periodic.go:216] Running conditional gatherer I0428 15:37:57.071545 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules I0428 15:37:57.078301 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.9:34610->172.30.0.10:53: read: connection refused E0428 15:37:57.078553 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0428 15:37:57.078610 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0428 15:37:57.084599 1 conditional_gatherer.go:392] cluster version is '4.21.9' E0428 15:37:57.084611 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 15:37:57.084616 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 15:37:57.084619 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 15:37:57.084623 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 15:37:57.084625 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 15:37:57.084629 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 15:37:57.084632 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 15:37:57.084634 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0428 15:37:57.084637 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0428 15:37:57.084650 1 tasks_processing.go:45] number of workers: 3 I0428 15:37:57.084661 1 tasks_processing.go:69] worker 2 listening for tasks. I0428 15:37:57.084667 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0428 15:37:57.084674 1 tasks_processing.go:69] worker 0 listening for tasks. I0428 15:37:57.084686 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0428 15:37:57.084687 1 tasks_processing.go:69] worker 1 listening for tasks. I0428 15:37:57.084691 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0428 15:37:57.084695 1 tasks_processing.go:74] worker 1 stopped. I0428 15:37:57.084749 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0428 15:37:57.084762 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.045µs to process 1 records I0428 15:37:57.084813 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0428 15:37:57.084822 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.144µs to process 1 records I0428 15:37:57.084827 1 tasks_processing.go:74] worker 0 stopped. I0428 15:37:57.084957 1 tasks_processing.go:74] worker 2 stopped. I0428 15:37:57.084969 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 258.434µs to process 0 records I0428 15:37:57.084994 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.21.9/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.9:34610->172.30.0.10:53: read: connection refused I0428 15:37:57.085011 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0428 15:37:57.108173 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=32767135ad01dd478a12b5b190c585bcac60c2d3058aee9a352fbe633c9d4456 I0428 15:37:57.108302 1 diskrecorder.go:70] Writing 101 records to /var/lib/insights-operator/insights-2026-04-28-153757.tar.gz I0428 15:37:57.113996 1 diskrecorder.go:51] Wrote 101 records to disk in 5ms I0428 15:37:57.114025 1 periodic.go:285] Gathering cluster info every 2h0m0s I0428 15:37:57.114041 1 periodic.go:286] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0428 15:38:04.060501 1 configmapobserver.go:84] configmaps "insights-config" not found I0428 15:38:50.594367 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="8489b1448879a811f9c0e3acc4e4663bf303b9c9aa17ebad36e75f6368e21213") W0428 15:38:50.594403 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0428 15:38:50.594465 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="4127628f00119091771bf5aa88bd491b38a2c1ff08d67cb780dc120df5f301a6") I0428 15:38:50.594478 1 base_controller.go:181] Shutting down ConfigController ... I0428 15:38:50.594484 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0428 15:38:50.594504 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0428 15:38:50.594523 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0428 15:38:50.594519 1 base_controller.go:181] Shutting down LoggingSyncer ... I0428 15:38:50.594549 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0428 15:38:50.594547 1 periodic.go:177] Shutting down I0428 15:38:50.594551 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0428 15:38:50.594551 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="f84ae05eccb62ba8134a1c4df89b0ae37d99ba323b088d2f65c2e030a03cd7d8") I0428 15:38:50.594564 1 genericapiserver.go:651] "[graceful-termination] not going to wait for active watch request(s) to drain" I0428 15:38:50.594564 1 base_controller.go:113] All LoggingSyncer workers have been terminated E0428 15:38:50.594597 1 controller.go:341] Unable to write cluster operator status: client rate limiter Wait returned an error: context canceled E0428 15:38:50.594631 1 controller.go:341] Unable to write cluster operator status: client rate limiter Wait returned an error: context canceled I0428 15:38:50.594638 1 secure_serving.go:258] Stopped listening on [::]:8443