W0506 07:38:12.854899 1 cmd.go:257] Using insecure, self-signed certificates I0506 07:38:13.700531 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0506 07:38:13.700851 1 observer_polling.go:159] Starting file observer I0506 07:38:14.473989 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0506 07:38:14.474179 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0506 07:38:14.474738 1 secure_serving.go:57] Forcing use of http/1.1 only W0506 07:38:14.474761 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0506 07:38:14.474765 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0506 07:38:14.474769 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0506 07:38:14.474771 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0506 07:38:14.474773 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0506 07:38:14.474776 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0506 07:38:14.474791 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0506 07:38:14.478274 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0506 07:38:14.478331 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"2e7912aa-6f4d-4a9c-b5d1-1eb0b031e600", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0506 07:38:14.479083 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0506 07:38:14.479102 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0506 07:38:14.479131 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0506 07:38:14.479143 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0506 07:38:14.479142 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0506 07:38:14.479153 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0506 07:38:14.479380 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-1775567394/tls.crt::/tmp/serving-cert-1775567394/tls.key" I0506 07:38:14.479702 1 secure_serving.go:213] Serving securely on [::]:8443 I0506 07:38:14.479741 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0506 07:38:14.483961 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0506 07:38:14.483992 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0506 07:38:14.484095 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0506 07:38:14.488337 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0506 07:38:14.488359 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0506 07:38:14.492334 1 secretconfigobserver.go:119] support secret does not exist I0506 07:38:14.496173 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0506 07:38:14.500191 1 secretconfigobserver.go:119] support secret does not exist I0506 07:38:14.505076 1 recorder.go:161] Pruning old reports every 6h32m42s, max age is 288h0m0s I0506 07:38:14.510937 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0506 07:38:14.510950 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0506 07:38:14.510960 1 insightsreport.go:296] Starting report retriever I0506 07:38:14.510958 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0506 07:38:14.510963 1 periodic.go:209] Running clusterconfig gatherer I0506 07:38:14.510966 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0506 07:38:14.511000 1 tasks_processing.go:45] number of workers: 64 I0506 07:38:14.511044 1 tasks_processing.go:69] worker 20 listening for tasks. I0506 07:38:14.511053 1 tasks_processing.go:69] worker 37 listening for tasks. I0506 07:38:14.511056 1 tasks_processing.go:71] worker 20 working on sap_datahubs task. I0506 07:38:14.511061 1 tasks_processing.go:69] worker 21 listening for tasks. I0506 07:38:14.511056 1 tasks_processing.go:69] worker 63 listening for tasks. I0506 07:38:14.511071 1 tasks_processing.go:69] worker 51 listening for tasks. I0506 07:38:14.511072 1 tasks_processing.go:69] worker 50 listening for tasks. I0506 07:38:14.511082 1 tasks_processing.go:69] worker 43 listening for tasks. I0506 07:38:14.511092 1 tasks_processing.go:69] worker 44 listening for tasks. I0506 07:38:14.511106 1 tasks_processing.go:69] worker 45 listening for tasks. I0506 07:38:14.511118 1 tasks_processing.go:69] worker 48 listening for tasks. I0506 07:38:14.511148 1 tasks_processing.go:69] worker 22 listening for tasks. I0506 07:38:14.511158 1 tasks_processing.go:69] worker 23 listening for tasks. I0506 07:38:14.511163 1 tasks_processing.go:69] worker 24 listening for tasks. I0506 07:38:14.511168 1 tasks_processing.go:69] worker 25 listening for tasks. I0506 07:38:14.511164 1 tasks_processing.go:69] worker 6 listening for tasks. I0506 07:38:14.511174 1 tasks_processing.go:69] worker 49 listening for tasks. I0506 07:38:14.511168 1 tasks_processing.go:69] worker 4 listening for tasks. I0506 07:38:14.511182 1 tasks_processing.go:69] worker 27 listening for tasks. I0506 07:38:14.511177 1 tasks_processing.go:69] worker 26 listening for tasks. I0506 07:38:14.511188 1 tasks_processing.go:69] worker 7 listening for tasks. I0506 07:38:14.511189 1 tasks_processing.go:69] worker 3 listening for tasks. I0506 07:38:14.511195 1 tasks_processing.go:69] worker 28 listening for tasks. I0506 07:38:14.511181 1 tasks_processing.go:69] worker 2 listening for tasks. I0506 07:38:14.511201 1 tasks_processing.go:69] worker 38 listening for tasks. I0506 07:38:14.511199 1 tasks_processing.go:69] worker 47 listening for tasks. I0506 07:38:14.511210 1 tasks_processing.go:69] worker 18 listening for tasks. I0506 07:38:14.511211 1 tasks_processing.go:69] worker 30 listening for tasks. I0506 07:38:14.511212 1 tasks_processing.go:69] worker 0 listening for tasks. I0506 07:38:14.511211 1 tasks_processing.go:69] worker 9 listening for tasks. I0506 07:38:14.511222 1 tasks_processing.go:69] worker 13 listening for tasks. I0506 07:38:14.511202 1 tasks_processing.go:69] worker 53 listening for tasks. I0506 07:38:14.511228 1 tasks_processing.go:69] worker 39 listening for tasks. I0506 07:38:14.511230 1 tasks_processing.go:69] worker 41 listening for tasks. I0506 07:38:14.511232 1 tasks_processing.go:69] worker 34 listening for tasks. I0506 07:38:14.511233 1 tasks_processing.go:69] worker 19 listening for tasks. I0506 07:38:14.511234 1 tasks_processing.go:69] worker 33 listening for tasks. I0506 07:38:14.511237 1 tasks_processing.go:69] worker 35 listening for tasks. I0506 07:38:14.511240 1 tasks_processing.go:69] worker 36 listening for tasks. I0506 07:38:14.511240 1 tasks_processing.go:69] worker 42 listening for tasks. I0506 07:38:14.511235 1 tasks_processing.go:69] worker 54 listening for tasks. I0506 07:38:14.511220 1 tasks_processing.go:69] worker 60 listening for tasks. I0506 07:38:14.511200 1 tasks_processing.go:69] worker 17 listening for tasks. I0506 07:38:14.511186 1 tasks_processing.go:69] worker 14 listening for tasks. I0506 07:38:14.511190 1 tasks_processing.go:69] worker 46 listening for tasks. I0506 07:38:14.511195 1 tasks_processing.go:69] worker 52 listening for tasks. I0506 07:38:14.511197 1 tasks_processing.go:69] worker 57 listening for tasks. I0506 07:38:14.511181 1 tasks_processing.go:69] worker 5 listening for tasks. I0506 07:38:14.511198 1 tasks_processing.go:69] worker 1 listening for tasks. I0506 07:38:14.511204 1 tasks_processing.go:69] worker 29 listening for tasks. I0506 07:38:14.511203 1 tasks_processing.go:69] worker 58 listening for tasks. I0506 07:38:14.511199 1 tasks_processing.go:69] worker 10 listening for tasks. I0506 07:38:14.511208 1 tasks_processing.go:69] worker 15 listening for tasks. I0506 07:38:14.511211 1 tasks_processing.go:69] worker 59 listening for tasks. I0506 07:38:14.511213 1 tasks_processing.go:69] worker 11 listening for tasks. I0506 07:38:14.511218 1 tasks_processing.go:69] worker 55 listening for tasks. I0506 07:38:14.511218 1 tasks_processing.go:69] worker 16 listening for tasks. I0506 07:38:14.511222 1 tasks_processing.go:69] worker 31 listening for tasks. I0506 07:38:14.511221 1 tasks_processing.go:69] worker 12 listening for tasks. I0506 07:38:14.511216 1 tasks_processing.go:69] worker 8 listening for tasks. I0506 07:38:14.511306 1 tasks_processing.go:71] worker 43 working on machine_healthchecks task. I0506 07:38:14.511311 1 tasks_processing.go:71] worker 21 working on jaegers task. I0506 07:38:14.511314 1 tasks_processing.go:71] worker 39 working on ingress_certificates task. I0506 07:38:14.511316 1 tasks_processing.go:71] worker 8 working on monitoring_persistent_volumes task. I0506 07:38:14.511319 1 tasks_processing.go:71] worker 44 working on operators_pods_and_events task. I0506 07:38:14.511324 1 tasks_processing.go:71] worker 18 working on machine_sets task. I0506 07:38:14.511321 1 tasks_processing.go:71] worker 50 working on validating_webhook_configurations task. I0506 07:38:14.511332 1 tasks_processing.go:71] worker 33 working on container_runtime_configs task. I0506 07:38:14.511332 1 tasks_processing.go:71] worker 9 working on machine_configs task. I0506 07:38:14.511347 1 tasks_processing.go:71] worker 29 working on authentication task. I0506 07:38:14.511355 1 tasks_processing.go:71] worker 30 working on nodenetworkconfigurationpolicies task. I0506 07:38:14.511378 1 tasks_processing.go:71] worker 53 working on feature_gates task. I0506 07:38:14.511306 1 tasks_processing.go:71] worker 63 working on cost_management_metrics_configs task. I0506 07:38:14.511389 1 tasks_processing.go:71] worker 54 working on config_maps task. I0506 07:38:14.511228 1 tasks_processing.go:69] worker 61 listening for tasks. I0506 07:38:14.511401 1 tasks_processing.go:71] worker 36 working on openshift_logging task. I0506 07:38:14.511405 1 tasks_processing.go:71] worker 11 working on openstack_controlplanes task. I0506 07:38:14.511407 1 tasks_processing.go:71] worker 61 working on mutating_webhook_configurations task. I0506 07:38:14.511415 1 tasks_processing.go:71] worker 7 working on storage_cluster task. I0506 07:38:14.511583 1 tasks_processing.go:71] worker 5 working on pdbs task. I0506 07:38:14.511713 1 tasks_processing.go:71] worker 10 working on machine_autoscalers task. I0506 07:38:14.511751 1 tasks_processing.go:71] worker 16 working on dvo_metrics task. I0506 07:38:14.511311 1 tasks_processing.go:71] worker 37 working on openshift_machine_api_events task. I0506 07:38:14.511820 1 tasks_processing.go:71] worker 60 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0506 07:38:14.511860 1 tasks_processing.go:71] worker 17 working on overlapping_namespace_uids task. I0506 07:38:14.511867 1 tasks_processing.go:71] worker 1 working on networks task. I0506 07:38:14.511872 1 tasks_processing.go:71] worker 52 working on certificate_signing_requests task. I0506 07:38:14.511879 1 tasks_processing.go:71] worker 48 working on pod_network_connectivity_checks task. I0506 07:38:14.511928 1 tasks_processing.go:71] worker 57 working on clusterroles task. I0506 07:38:14.512109 1 tasks_processing.go:71] worker 3 working on openstack_dataplanedeployments task. I0506 07:38:14.512246 1 tasks_processing.go:71] worker 14 working on container_images task. I0506 07:38:14.511875 1 tasks_processing.go:71] worker 45 working on ceph_cluster task. I0506 07:38:14.511234 1 tasks_processing.go:69] worker 40 listening for tasks. I0506 07:38:14.511228 1 tasks_processing.go:69] worker 32 listening for tasks. I0506 07:38:14.511306 1 tasks_processing.go:71] worker 51 working on nodes task. I0506 07:38:14.512715 1 tasks_processing.go:71] worker 55 working on silenced_alerts task. I0506 07:38:14.511325 1 tasks_processing.go:71] worker 34 working on infrastructures task. I0506 07:38:14.511315 1 tasks_processing.go:71] worker 42 working on qemu_kubevirt_launcher_logs task. I0506 07:38:14.511320 1 tasks_processing.go:71] worker 0 working on machines task. I0506 07:38:14.511321 1 tasks_processing.go:71] worker 41 working on storage_classes task. I0506 07:38:14.511327 1 tasks_processing.go:71] worker 13 working on active_alerts task. I0506 07:38:14.511329 1 tasks_processing.go:71] worker 19 working on oauths task. I0506 07:38:14.511867 1 tasks_processing.go:71] worker 49 working on schedulers task. I0506 07:38:14.511385 1 tasks_processing.go:71] worker 46 working on nodenetworkstates task. I0506 07:38:14.511399 1 tasks_processing.go:71] worker 35 working on sap_pods task. I0506 07:38:14.512727 1 tasks_processing.go:71] worker 58 working on openstack_version task. I0506 07:38:14.513093 1 tasks_processing.go:71] worker 22 working on image task. I0506 07:38:14.512733 1 tasks_processing.go:71] worker 31 working on support_secret task. I0506 07:38:14.513339 1 tasks_processing.go:71] worker 27 working on lokistack task. I0506 07:38:14.513344 1 tasks_processing.go:71] worker 38 working on operators task. I0506 07:38:14.513349 1 tasks_processing.go:71] worker 59 working on node_logs task. I0506 07:38:14.511232 1 tasks_processing.go:69] worker 56 listening for tasks. I0506 07:38:14.512741 1 tasks_processing.go:71] worker 2 working on ingress task. I0506 07:38:14.513509 1 tasks_processing.go:71] worker 23 working on metrics task. W0506 07:38:14.513540 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0506 07:38:14.513578 1 tasks_processing.go:71] worker 23 working on sap_config task. I0506 07:38:14.513607 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 33.8µs to process 0 records I0506 07:38:14.513680 1 tasks_processing.go:71] worker 24 working on cluster_apiserver task. I0506 07:38:14.513692 1 tasks_processing.go:71] worker 26 working on install_plans task. I0506 07:38:14.512745 1 tasks_processing.go:71] worker 28 working on crds task. I0506 07:38:14.513994 1 tasks_processing.go:71] worker 6 working on aggregated_monitoring_cr_names task. I0506 07:38:14.514061 1 tasks_processing.go:71] worker 40 working on proxies task. W0506 07:38:14.514166 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0506 07:38:14.514228 1 tasks_processing.go:71] worker 13 working on service_accounts task. I0506 07:38:14.512738 1 tasks_processing.go:71] worker 15 working on openstack_dataplanenodesets task. I0506 07:38:14.513943 1 tasks_processing.go:71] worker 4 working on tsdb_status task. W0506 07:38:14.514338 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory W0506 07:38:14.514427 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0506 07:38:14.514498 1 tasks_processing.go:71] worker 32 working on image_registries task. I0506 07:38:14.514541 1 tasks_processing.go:71] worker 56 working on olm_operators task. I0506 07:38:14.512736 1 tasks_processing.go:71] worker 12 working on version task. I0506 07:38:14.513947 1 tasks_processing.go:71] worker 25 working on image_pruners task. I0506 07:38:14.511313 1 tasks_processing.go:71] worker 47 working on machine_config_pools task. I0506 07:38:14.514811 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 116.765µs to process 0 records I0506 07:38:14.514828 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 30.489µs to process 0 records I0506 07:38:14.514841 1 tasks_processing.go:74] worker 4 stopped. I0506 07:38:14.514847 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 124.095µs to process 0 records I0506 07:38:14.514856 1 tasks_processing.go:74] worker 55 stopped. I0506 07:38:14.511230 1 tasks_processing.go:69] worker 62 listening for tasks. I0506 07:38:14.514866 1 tasks_processing.go:74] worker 62 stopped. I0506 07:38:14.515698 1 tasks_processing.go:74] worker 36 stopped. I0506 07:38:14.515715 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 4.28754ms to process 0 records I0506 07:38:14.515750 1 tasks_processing.go:74] worker 7 stopped. I0506 07:38:14.515767 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 4.323371ms to process 0 records I0506 07:38:14.515802 1 tasks_processing.go:74] worker 20 stopped. I0506 07:38:14.515816 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 4.734893ms to process 0 records I0506 07:38:14.515901 1 tasks_processing.go:74] worker 33 stopped. I0506 07:38:14.515932 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 4.557381ms to process 0 records I0506 07:38:14.518838 1 tasks_processing.go:74] worker 45 stopped. I0506 07:38:14.518854 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 6.427195ms to process 0 records I0506 07:38:14.526004 1 tasks_processing.go:74] worker 18 stopped. I0506 07:38:14.526022 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 14.667358ms to process 0 records I0506 07:38:14.526033 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 11.921697ms to process 0 records I0506 07:38:14.526040 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 13.011248ms to process 0 records I0506 07:38:14.526048 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 13.919563ms to process 0 records I0506 07:38:14.526048 1 tasks_processing.go:74] worker 46 stopped. I0506 07:38:14.526054 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 14.68486ms to process 0 records I0506 07:38:14.526058 1 tasks_processing.go:74] worker 30 stopped. I0506 07:38:14.526061 1 tasks_processing.go:74] worker 58 stopped. I0506 07:38:14.526064 1 tasks_processing.go:74] worker 3 stopped. I0506 07:38:14.526072 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 12.695295ms to process 0 records E0506 07:38:14.526084 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0506 07:38:14.526095 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 14.173434ms to process 0 records I0506 07:38:14.526104 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 14.741396ms to process 0 records I0506 07:38:14.526110 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 11.764366ms to process 0 records I0506 07:38:14.526116 1 tasks_processing.go:74] worker 15 stopped. I0506 07:38:14.526122 1 tasks_processing.go:74] worker 27 stopped. I0506 07:38:14.526127 1 tasks_processing.go:74] worker 48 stopped. I0506 07:38:14.526132 1 tasks_processing.go:74] worker 21 stopped. I0506 07:38:14.526226 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0506 07:38:14.526244 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0506 07:38:14.526248 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0506 07:38:14.526251 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0506 07:38:14.526269 1 controller.go:489] The operator is still being initialized I0506 07:38:14.526277 1 controller.go:512] The operator is healthy I0506 07:38:14.528485 1 tasks_processing.go:74] worker 11 stopped. I0506 07:38:14.528498 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 17.071294ms to process 0 records I0506 07:38:14.530769 1 tasks_processing.go:74] worker 63 stopped. I0506 07:38:14.530780 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 19.375025ms to process 0 records I0506 07:38:14.530785 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 19.045699ms to process 0 records I0506 07:38:14.530791 1 tasks_processing.go:74] worker 10 stopped. I0506 07:38:14.530919 1 tasks_processing.go:74] worker 43 stopped. E0506 07:38:14.530935 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0506 07:38:14.530948 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 19.603318ms to process 0 records I0506 07:38:14.530960 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 17.34061ms to process 0 records I0506 07:38:14.530969 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 16.522747ms to process 0 records I0506 07:38:14.530978 1 tasks_processing.go:74] worker 35 stopped. I0506 07:38:14.530984 1 tasks_processing.go:74] worker 23 stopped. I0506 07:38:14.531219 1 tasks_processing.go:74] worker 29 stopped. I0506 07:38:14.531458 1 recorder.go:75] Recording config/authentication with fingerprint=8bd3937bc40ab8e012c9f368fd918566a3fa9839f6a0eb9dfcc15c58a9eecac0 I0506 07:38:14.531476 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 19.852982ms to process 1 records I0506 07:38:14.531559 1 tasks_processing.go:74] worker 53 stopped. I0506 07:38:14.531580 1 recorder.go:75] Recording config/featuregate with fingerprint=fa11b23423e68d497dab530093d8386edd6ac231fd57cd19365a550fc515fdfb I0506 07:38:14.531590 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 19.829546ms to process 1 records I0506 07:38:14.536160 1 tasks_processing.go:74] worker 31 stopped. E0506 07:38:14.536172 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0506 07:38:14.536186 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 22.954067ms to process 0 records E0506 07:38:14.536208 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0506 07:38:14.536217 1 gather.go:177] gatherer "clusterconfig" function "machines" took 21.388089ms to process 0 records I0506 07:38:14.536302 1 recorder.go:75] Recording config/proxy with fingerprint=cb10bdab5be5edf3c2b471dde45e6fb80d80024b411ab8d7b38cc2ad14ffff54 I0506 07:38:14.536305 1 tasks_processing.go:74] worker 0 stopped. I0506 07:38:14.536315 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 22.07418ms to process 1 records I0506 07:38:14.536324 1 tasks_processing.go:74] worker 40 stopped. I0506 07:38:14.536323 1 tasks_processing.go:74] worker 22 stopped. I0506 07:38:14.536394 1 recorder.go:75] Recording config/image with fingerprint=88ac5eba1cd7b7119c589602f7920eabbbe0b4a8ed809212f66e1ee5b2033c89 I0506 07:38:14.536409 1 gather.go:177] gatherer "clusterconfig" function "image" took 23.214252ms to process 1 records I0506 07:38:14.542656 1 tasks_processing.go:74] worker 2 stopped. I0506 07:38:14.542837 1 recorder.go:75] Recording config/ingress with fingerprint=7de2264c95f6bd67c728629229b00a21cb4fc1c72000258a645c36f559f54860 I0506 07:38:14.542857 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 29.149576ms to process 1 records I0506 07:38:14.542867 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 31.427977ms to process 0 records I0506 07:38:14.542885 1 tasks_processing.go:74] worker 8 stopped. I0506 07:38:14.542919 1 tasks_processing.go:74] worker 24 stopped. I0506 07:38:14.543024 1 recorder.go:75] Recording config/apiserver with fingerprint=4b5eab8a2258d61c8448b206b74afe818530c7f8d1274ec20f09466222c6dc8b I0506 07:38:14.543037 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 29.070208ms to process 1 records I0506 07:38:14.543049 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 30.978636ms to process 0 records I0506 07:38:14.543127 1 tasks_processing.go:74] worker 37 stopped. I0506 07:38:14.543190 1 tasks_processing.go:74] worker 34 stopped. I0506 07:38:14.543528 1 recorder.go:75] Recording config/infrastructure with fingerprint=92f81bdfea01b800bedd9c3ec8b7b093ab53a96b2f6537cb32de4731e8742491 I0506 07:38:14.543542 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 28.536338ms to process 1 records I0506 07:38:14.543625 1 tasks_processing.go:74] worker 25 stopped. I0506 07:38:14.543767 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=5c7f0876caac163e3c13c62c8d4fd86ef8ad28fcdbadcaec38c3fbdfdb22cc86 I0506 07:38:14.543781 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 28.375717ms to process 1 records I0506 07:38:14.543860 1 tasks_processing.go:74] worker 1 stopped. I0506 07:38:14.543900 1 recorder.go:75] Recording config/network with fingerprint=7fd8eb37d6a08886ffb01102de2754fb4c97bbab2a3997ccf4f2fc7b5029c40d I0506 07:38:14.543931 1 gather.go:177] gatherer "clusterconfig" function "networks" took 31.314493ms to process 1 records I0506 07:38:14.544012 1 tasks_processing.go:74] worker 19 stopped. I0506 07:38:14.544118 1 recorder.go:75] Recording config/oauth with fingerprint=9756fc1ebb39ce68e927f904631b3cd43fb99fa8412b7fb7009ae7afb6aa21f4 I0506 07:38:14.544131 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 28.407852ms to process 1 records I0506 07:38:14.546923 1 tasks_processing.go:74] worker 32 stopped. I0506 07:38:14.547520 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=dd9b99ac66bdff1ebb1405a9d99c84144976c7d728920ba2d91136e2bebc306b I0506 07:38:14.547540 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 32.361146ms to process 1 records I0506 07:38:14.547625 1 tasks_processing.go:74] worker 5 stopped. I0506 07:38:14.547646 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=99cf7ab73141b8d7c31f195f46493d2d96fba3d26d61c5751ff43ef5f9e76d07 I0506 07:38:14.547671 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=f3fcf15951c731fad96ac569ea6106b21949bca69f3c2adad2709f3451f9fb4f I0506 07:38:14.547698 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=c454ac6c64c3d4cd32577353bb1e12e51ec0eb36689679f3ae4f60b99629e03d I0506 07:38:14.547707 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 35.722722ms to process 3 records I0506 07:38:14.547782 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=b610247399c413ca748eafd5dd7cb09886d98d94195b343c2c84aea9fa12e71b I0506 07:38:14.547786 1 tasks_processing.go:74] worker 49 stopped. I0506 07:38:14.547800 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 33.446622ms to process 1 records I0506 07:38:14.553616 1 tasks_processing.go:74] worker 41 stopped. I0506 07:38:14.553741 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=87858159e4ef09c7a97ae59d6fc5729161465768faf3ad8e8e42d545c8a1227d I0506 07:38:14.553774 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=978d58a4e9bc6f76e710403d65c5a2fb0c67d62513163a22e7341bb261f57694 I0506 07:38:14.553786 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 40.109523ms to process 2 records I0506 07:38:14.553831 1 gather_logs.go:145] no pods in namespace were found I0506 07:38:14.553851 1 tasks_processing.go:74] worker 42 stopped. I0506 07:38:14.553862 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 39.192882ms to process 0 records I0506 07:38:14.553924 1 tasks_processing.go:74] worker 51 stopped. I0506 07:38:14.554203 1 recorder.go:75] Recording config/node/ip-10-0-1-114.ec2.internal with fingerprint=e416f0ffd10bbbbabe292b6a876d8c0868c6a8d4b6808cc764d0a1afd4a67476 I0506 07:38:14.554221 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 41.2203ms to process 1 records I0506 07:38:14.554385 1 tasks_processing.go:74] worker 59 stopped. I0506 07:38:14.554404 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 41.021241ms to process 0 records I0506 07:38:14.554730 1 tasks_processing.go:74] worker 52 stopped. I0506 07:38:14.554747 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 42.84676ms to process 0 records I0506 07:38:14.556720 1 tasks_processing.go:74] worker 61 stopped. I0506 07:38:14.556856 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s I0506 07:38:14.556930 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s W0506 07:38:14.557552 1 operator.go:288] started I0506 07:38:14.557670 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0506 07:38:14.558179 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=a34f3c36665ae141b76c9cbb121856d266f3b558428858dcc0b7213ca6f47ddb I0506 07:38:14.558246 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=1103fe308061146f99905a4a1b923a58bd7c22022d383a20ec2636f6b871665a I0506 07:38:14.558333 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=53145199ddab17d08e853954aff4090c5cdcc6f7cffa6f930dfb88e5af4d48a7 I0506 07:38:14.558345 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 45.283789ms to process 3 records I0506 07:38:14.558509 1 tasks_processing.go:74] worker 50 stopped. I0506 07:38:14.558780 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=5e13f5ee3e1b902989f0854ac9bd3eee0c019b9b15e65fd8669aa9df7d7ad814 W0506 07:38:14.559034 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0506 07:38:14.559352 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=0d2af4da72d3c82431db2815be8e21f241d3d97e53c1d6b569a01b857dc2673d I0506 07:38:14.559543 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=b243723ab1d33c3b0123bd2ace0d01b4ad90f43fa5e93a0ee126d169a79bbd2e I0506 07:38:14.559589 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=ba36767946764478fbb1a58e399aac498256528bb68e89be19d67922761c3a90 I0506 07:38:14.559635 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=b7e733b34d1689ca314967afdca5b7e8fd0c3599aff8771f8b0a40821736ac8e I0506 07:38:14.559663 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=3f788ae1f79292719548391e24a9a3ea5dc82b4becaafa7b33595ca0697d0547 I0506 07:38:14.560482 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=10e92b81415e68c01e488ff10348ad576810a1975545fac61139fd58d21630dc I0506 07:38:14.560532 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=c694b1ea13adce057b0269a9f595bea9b12150acfc78b29f9fcf40db6159dcbf I0506 07:38:14.560569 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=218156ae2b3d62e275612f32ea8d6c80942e6e372f0551c05df3602cc2bca637 I0506 07:38:14.560594 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=e0148b007ecff7c5bf6c4f383c8d078862f42b5f0f42c0edaa16b7d9587c031d I0506 07:38:14.560619 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=69af4b350385c59d0fa58da40f01489852180e93aede20cc28763baaf346bc65 I0506 07:38:14.560630 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 47.167013ms to process 11 records I0506 07:38:14.560724 1 tasks_processing.go:74] worker 28 stopped. I0506 07:38:14.561271 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=5f187dd5d86719e7aa6b63ad7e320add9fd0db17ea97b15857edce0cb29fe757 I0506 07:38:14.561471 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=3af813404ec59a0c20ba080d102c1fddb06dc9de48afc13c2c0bf049c2594c80 I0506 07:38:14.561481 1 gather.go:177] gatherer "clusterconfig" function "crds" took 44.953269ms to process 2 records I0506 07:38:14.561563 1 tasks_processing.go:74] worker 57 stopped. I0506 07:38:14.561617 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=ce0b931517b50f33e4c0e04f8c4964c1bb560e44f2becdf27b0d0e4f3d91c9b4 I0506 07:38:14.561696 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=5082ea8172218cd9388047bafb26db283586d78f6c74f344de039f34fd73f5a4 I0506 07:38:14.561703 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 48.509162ms to process 2 records I0506 07:38:14.561717 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0506 07:38:14.561725 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 49.16866ms to process 1 records I0506 07:38:14.561731 1 tasks_processing.go:74] worker 17 stopped. I0506 07:38:14.567890 1 tasks_processing.go:74] worker 6 stopped. I0506 07:38:14.567945 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 53.869952ms to process 0 records I0506 07:38:14.568037 1 tasks_processing.go:74] worker 56 stopped. I0506 07:38:14.568134 1 recorder.go:75] Recording config/olm_operators with fingerprint=20aa688d782649ad550a8fcc2a28d7a0fbb5f38df1c7f00eccf17a7e4e20074f I0506 07:38:14.568161 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 53.468723ms to process 1 records I0506 07:38:14.570811 1 tasks_processing.go:74] worker 14 stopped. I0506 07:38:14.572108 1 recorder.go:75] Recording config/pod/openshift-console-operator/console-operator-575cd97545-qfjm8 with fingerprint=57395f5ad143e2d6566863782221feb75bb68ff09606f4fda115c3e7833777c1 I0506 07:38:14.572166 1 recorder.go:75] Recording config/running_containers with fingerprint=4d2ec7c16b4d54920310d451bfa031ed33b3763840e8017abdc0430c5f789dd0 I0506 07:38:14.572177 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 58.542549ms to process 2 records I0506 07:38:14.579394 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0506 07:38:14.579408 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0506 07:38:14.579418 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0506 07:38:14.584134 1 base_controller.go:82] Caches are synced for ConfigController I0506 07:38:14.584156 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... I0506 07:38:14.585358 1 prometheus_rules.go:88] Prometheus rules successfully created I0506 07:38:14.585362 1 tasks_processing.go:74] worker 60 stopped. I0506 07:38:14.585388 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 73.529875ms to process 0 records I0506 07:38:14.585593 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0506 07:38:14.585604 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0506 07:38:14.585607 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0506 07:38:14.585611 1 controller.go:212] Source scaController *sca.Controller is not ready I0506 07:38:14.585615 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0506 07:38:14.585632 1 controller.go:489] The operator is still being initialized I0506 07:38:14.585640 1 controller.go:512] The operator is healthy E0506 07:38:14.586528 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27171218f5-c999-4bd9-8fd0-429396fc333d%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:40037->172.30.0.10:53: read: connection refused I0506 07:38:14.586541 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%27171218f5-c999-4bd9-8fd0-429396fc333d%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:40037->172.30.0.10:53: read: connection refused I0506 07:38:14.595221 1 configmapobserver.go:84] configmaps "insights-config" not found I0506 07:38:14.609099 1 tasks_processing.go:74] worker 39 stopped. E0506 07:38:14.609126 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0506 07:38:14.609134 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2q49pdbn2k5bs7emt873shpqpjujrkvp-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2q49pdbn2k5bs7emt873shpqpjujrkvp-primary-cert-bundle-secret" not found I0506 07:38:14.609227 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=507f2e291311573c3bd619e397b078e0f076cff42f188f50491c14fe4fcdef16 I0506 07:38:14.609245 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 97.762949ms to process 1 records I0506 07:38:14.615700 1 tasks_processing.go:74] worker 54 stopped. E0506 07:38:14.615718 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0506 07:38:14.615724 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0506 07:38:14.615728 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0506 07:38:14.615739 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0506 07:38:14.615766 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0506 07:38:14.615773 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0506 07:38:14.615777 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0506 07:38:14.615781 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0506 07:38:14.615818 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0506 07:38:14.615826 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0506 07:38:14.615831 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 104.292726ms to process 7 records I0506 07:38:14.644097 1 tasks_processing.go:74] worker 12 stopped. I0506 07:38:14.644516 1 recorder.go:75] Recording config/version with fingerprint=b508f813676e79c16035f9c899e367c4b7b91d227f41ef387d62dcaf45e7753a I0506 07:38:14.644537 1 recorder.go:75] Recording config/id with fingerprint=6ff9515250ec808295eae5251b609452c57c5beb52cd073d4a0a1730fcf8710d I0506 07:38:14.644544 1 gather.go:177] gatherer "clusterconfig" function "version" took 129.387007ms to process 2 records I0506 07:38:14.650960 1 tasks_processing.go:74] worker 9 stopped. I0506 07:38:14.650992 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0506 07:38:14.651002 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 139.613485ms to process 1 records I0506 07:38:14.657847 1 base_controller.go:82] Caches are synced for LoggingSyncer I0506 07:38:14.657864 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0506 07:38:14.657969 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0506 07:38:14.661180 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0506 07:38:14.664857 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:36308->172.30.0.10:53: read: connection refused I0506 07:38:14.664872 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.13:36308->172.30.0.10:53: read: connection refused I0506 07:38:14.668136 1 gather_cluster_operators.go:184] Unable to get dnsrecords.ingress.operator.openshift.io resource due to: dnsrecords.ingress.operator.openshift.io "default" not found I0506 07:38:14.956931 1 gather_cluster_operator_pods_and_events.go:121] Found 10 pods with 11 containers I0506 07:38:14.956950 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 2287802 bytes I0506 07:38:14.957583 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-x687t pod in namespace openshift-dns (previous: false). I0506 07:38:15.180722 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-x687t pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-x687t\" is waiting to start: ContainerCreating" I0506 07:38:15.180745 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-x687t\" is waiting to start: ContainerCreating" I0506 07:38:15.180753 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-x687t pod in namespace openshift-dns (previous: false). I0506 07:38:15.360358 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-x687t pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-x687t\" is waiting to start: ContainerCreating" I0506 07:38:15.360460 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-x687t\" is waiting to start: ContainerCreating" I0506 07:38:15.360504 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-wf4mz pod in namespace openshift-dns (previous: false). W0506 07:38:15.557887 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0506 07:38:15.559061 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0506 07:38:15.559120 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7cdc5d6495-dwfsj pod in namespace openshift-image-registry (previous: false). I0506 07:38:15.562031 1 tasks_processing.go:74] worker 47 stopped. I0506 07:38:15.562046 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 1.047211235s to process 0 records I0506 07:38:15.754901 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0506 07:38:15.754965 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-7cdc5d6495-nxht7 pod in namespace openshift-image-registry (previous: false). I0506 07:38:15.954321 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0506 07:38:15.954379 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-d5c659444-4htdb pod in namespace openshift-image-registry (previous: false). I0506 07:38:15.986069 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0506 07:38:16.159748 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-d5c659444-4htdb pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-d5c659444-4htdb\" is waiting to start: ContainerCreating" I0506 07:38:16.159767 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-d5c659444-4htdb\" is waiting to start: ContainerCreating" I0506 07:38:16.159779 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-x829c pod in namespace openshift-image-registry (previous: false). I0506 07:38:16.360474 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0506 07:38:16.360494 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-544547969c-wbvnp pod in namespace openshift-ingress (previous: false). W0506 07:38:16.557811 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0506 07:38:16.559286 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-544547969c-wbvnp pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-544547969c-wbvnp\" is waiting to start: ContainerCreating" I0506 07:38:16.559302 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-544547969c-wbvnp\" is waiting to start: ContainerCreating" I0506 07:38:16.559315 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5b8cb84c9c-bmbdh pod in namespace openshift-ingress (previous: false). I0506 07:38:16.589031 1 tasks_processing.go:74] worker 38 stopped. I0506 07:38:16.589086 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=3f3c136393bc95e21c908cd9835484efe2ee606b5bfffcc8cc97d62ea0edabb2 I0506 07:38:16.589118 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=6f4f7d8fd22d23f4a3e77dd2487daac2a9d8a27fdac7c31c70486a7fadc645bb I0506 07:38:16.589152 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0506 07:38:16.589181 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=40a7aab8e20c8931cb5d96746482769198be634d79835ce84a596260431b5fbc I0506 07:38:16.589198 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0506 07:38:16.589221 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=16159b4823f3f76d306024517c52f6909db2174c695610ddff8893daba66762f I0506 07:38:16.589269 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=5b063e9566de724da90811e65a522b5ecf001477321eb98b64284feca77b1d47 I0506 07:38:16.589295 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=68793437129b674b6d65a6e38c6c84f154b964d9438d53873836920b13e9e86f I0506 07:38:16.589310 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=c584b600ec4ac1285af9910713565a831f773f6a4aea7b7ba18c0212c3e959df I0506 07:38:16.589327 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=3208414f898ca29f8b25ffc4e216bad305a7009e0d6c98f1a6231fa2ff4d8e7b I0506 07:38:16.589338 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0506 07:38:16.589354 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=9bc7bc2937cb07bd5d5a6f1a2c351a01c7c6066e4add72aec3b7629a1279f22c I0506 07:38:16.589365 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0506 07:38:16.589382 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=db094afdbb6c7a18f7b4170938b1cb8dd242d9d3c01ea14f22d5d4c023cef5bc I0506 07:38:16.589392 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0506 07:38:16.589424 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=b8505cf66fcd57075ba00c8cbdcb9bfb449a74c32a2df942dc8303b91ac83ac8 I0506 07:38:16.589435 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0506 07:38:16.589450 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=7f76cce4066cdc68a1cd23506a15c9295317330d261a0c5123c3c863bb32321c I0506 07:38:16.589579 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=5f8244b273ec917d6a18ddf0c45f417abe6bf6c699e1f52873730fb7542e3b8b I0506 07:38:16.589589 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0506 07:38:16.589596 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0506 07:38:16.589616 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0506 07:38:16.589637 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=b2f409b6440d6dbfa89ada091725d7bff6c00c8f0f7da98693ab3820cc643691 I0506 07:38:16.589662 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=f41cc5b394084b458a9afe9d9566021953a3129366c598bde3f1810be71bdd61 I0506 07:38:16.589671 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0506 07:38:16.589687 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=4ed073617826dd1febadac6f6c0b8c0a004d90384a6e7c604305a508c6105141 I0506 07:38:16.589696 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0506 07:38:16.589709 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=ef44b60e0438db024f6f6cbd56797f4d5c31e8e06c238318cba9453ea876e356 I0506 07:38:16.589724 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=636048044f6616f24c1f453311d3c779a8a83ba2506f4b837ef34a9457b9310e I0506 07:38:16.589739 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=7ba0cc1dd9238f87219d4360f103dfb79f7c96f3519ffd165767b1f58eea8902 I0506 07:38:16.589755 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=39339734b86bf959e441d0b54356b5cbb1bd00cf7e62f8b7c110a79c26b0b010 I0506 07:38:16.589777 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=5fc91f829934af8f34a0f0e88e6325ff3d102635edae59360529f632260447ac I0506 07:38:16.589787 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/serviceca/cluster with fingerprint=812f7edc2cdb30e61e7f2b29454357a40b1a507a4b0c2b7729193b67f0e3b4aa I0506 07:38:16.589815 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=d22300ecec31e20b21c30fe763dfe21bb65a0770f75b045a4e77dc346c773c43 I0506 07:38:16.589832 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0506 07:38:16.589840 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0506 07:38:16.589847 1 gather.go:177] gatherer "clusterconfig" function "operators" took 2.075662321s to process 36 records I0506 07:38:16.755435 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0506 07:38:16.755456 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-5b8cb84c9c-dl98v pod in namespace openshift-ingress (previous: false). I0506 07:38:16.959694 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-5b8cb84c9c-dl98v pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-5b8cb84c9c-dl98v\" is waiting to start: ContainerCreating" I0506 07:38:16.959715 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-5b8cb84c9c-dl98v\" is waiting to start: ContainerCreating" I0506 07:38:16.959729 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-r2cnc pod in namespace openshift-ingress-canary (previous: false). I0506 07:38:17.159691 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-r2cnc pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-r2cnc\" is waiting to start: ContainerCreating" I0506 07:38:17.159711 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-r2cnc\" is waiting to start: ContainerCreating" I0506 07:38:17.159729 1 tasks_processing.go:74] worker 44 stopped. I0506 07:38:17.159832 1 recorder.go:75] Recording events/openshift-dns-operator with fingerprint=a99922907fae1a0d4e65ddbdfc41b90979b4d66b7511c918a9a9016b305c5f39 I0506 07:38:17.159863 1 recorder.go:75] Recording events/openshift-dns with fingerprint=31affb7cb924d59668471ea6f3693fed571dc35b605e21599e67578b2a4af20e I0506 07:38:17.159949 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=a57daafb542ee06953df919de6d544a8d5322d20e0655b2c2da3241392748632 I0506 07:38:17.159976 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=1b8b17dfecf762e7cc894f5f0182ef9b1ac45e17f3f042e193c8c2d8a63ea02c I0506 07:38:17.160017 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=e191f756bb87beb5e5023bc59213057dcf867a89019c1f45dc0ac3eae0db7194 I0506 07:38:17.160029 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=ff287d2f87f3ea589624828175d22b7ae1712df9b100477485647c134904e555 I0506 07:38:17.160191 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-7cdc5d6495-dwfsj with fingerprint=dffeef4d9eb5b01b145b03264ab52b851b55c95ed553099474fb5c306dce5e74 I0506 07:38:17.160286 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-7cdc5d6495-nxht7 with fingerprint=26e913fd38d0f8c5bbadca178ff3d82eed41c9d3beb6f34fa95073db852f69eb I0506 07:38:17.160384 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-d5c659444-4htdb with fingerprint=894478eca27302013c99dbf2c6c23b5410838036f2e4a65adde4eb9e20af92e8 I0506 07:38:17.160393 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 2.648395278s to process 9 records W0506 07:38:17.557924 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0506 07:38:18.557393 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0506 07:38:19.553694 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0506 07:38:19.553727 1 tasks_processing.go:74] worker 16 stopped. E0506 07:38:19.553739 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0506 07:38:19.553751 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0506 07:38:19.553766 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0506 07:38:19.553776 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.041955901s to process 1 records I0506 07:38:26.892458 1 configmapobserver.go:84] configmaps "insights-config" not found I0506 07:38:26.966613 1 tasks_processing.go:74] worker 26 stopped. I0506 07:38:26.966660 1 recorder.go:75] Recording config/installplans with fingerprint=95dfd5f33a9a46199239158a0ec0183b629818eca24ae41f4d706d7a3d604aec I0506 07:38:26.966674 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.452898212s to process 1 records I0506 07:38:27.094579 1 configmapobserver.go:84] configmaps "insights-config" not found I0506 07:38:27.677941 1 configmapobserver.go:84] configmaps "insights-config" not found I0506 07:38:27.720755 1 tasks_processing.go:74] worker 13 stopped. I0506 07:38:27.721040 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=b3a2c4fa45ed7ae58b9c4093be9a56682fdfea23d7350a26feed0cd7c1d1f420 I0506 07:38:27.721056 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.206503133s to process 1 records E0506 07:38:27.721111 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.21s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"support_secret\" failed with an error, function \"machines\" failed with an error, function \"ingress_certificates\" failed with an error, function \"config_maps\" failed with an error, function \"dvo_metrics\" failed with an error" I0506 07:38:27.722217 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "support_secret" failed with an error, function "machines" failed with an error, function "ingress_certificates" failed with an error, function "config_maps" failed with an error, function "dvo_metrics" failed with an error I0506 07:38:27.722233 1 periodic.go:209] Running workloads gatherer I0506 07:38:27.722247 1 tasks_processing.go:45] number of workers: 2 I0506 07:38:27.722254 1 tasks_processing.go:69] worker 1 listening for tasks. I0506 07:38:27.722257 1 tasks_processing.go:71] worker 1 working on workload_info task. I0506 07:38:27.722267 1 tasks_processing.go:69] worker 0 listening for tasks. I0506 07:38:27.722286 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0506 07:38:27.742089 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 19s for image data I0506 07:38:27.749595 1 tasks_processing.go:74] worker 0 stopped. I0506 07:38:27.749615 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 27.289585ms to process 0 records I0506 07:38:27.754572 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (13ms) I0506 07:38:27.763599 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (9ms) I0506 07:38:27.771831 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (8ms) I0506 07:38:27.780504 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (9ms) I0506 07:38:27.788667 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (8ms) I0506 07:38:27.797669 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (9ms) I0506 07:38:27.805596 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (8ms) I0506 07:38:27.813776 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (8ms) I0506 07:38:27.822083 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (8ms) I0506 07:38:27.830238 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (8ms) I0506 07:38:27.850754 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (20ms) I0506 07:38:27.954158 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (103ms) I0506 07:38:28.058944 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (105ms) I0506 07:38:28.151407 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (92ms) I0506 07:38:28.252326 1 gather_workloads_info.go:387] No image sha256:ae7d3453fd734ecc865e5f9bb16f29244ebffe6291b77e1d4e496f71eb053174 (101ms) I0506 07:38:28.351814 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (99ms) I0506 07:38:28.451272 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (99ms) I0506 07:38:28.551254 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (100ms) I0506 07:38:28.651774 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (100ms) I0506 07:38:28.751105 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (99ms) I0506 07:38:28.850598 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (99ms) I0506 07:38:28.950962 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (100ms) I0506 07:38:29.056006 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (105ms) I0506 07:38:29.151721 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (96ms) I0506 07:38:29.250968 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (99ms) I0506 07:38:29.352151 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (101ms) I0506 07:38:29.451599 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (99ms) I0506 07:38:29.551673 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (100ms) I0506 07:38:29.651566 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (100ms) I0506 07:38:29.751827 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (100ms) I0506 07:38:29.851085 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (99ms) I0506 07:38:29.851115 1 tasks_processing.go:74] worker 1 stopped. E0506 07:38:29.851126 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0506 07:38:29.851463 1 recorder.go:75] Recording config/workload_info with fingerprint=0d66e6329940a6408933f68b84026217337e87fda07efd4a8f888a716ece882c I0506 07:38:29.851478 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.128850668s to process 1 records E0506 07:38:29.851502 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.129s with: function \"workload_info\" failed with an error" I0506 07:38:29.852613 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0506 07:38:29.852627 1 periodic.go:209] Running conditional gatherer I0506 07:38:29.859092 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0506 07:38:29.865530 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.13:53366->172.30.0.10:53: read: connection refused E0506 07:38:29.865789 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0506 07:38:29.865850 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0506 07:38:29.871710 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0506 07:38:29.871728 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 07:38:29.871733 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 07:38:29.871737 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 07:38:29.871741 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 07:38:29.871744 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 07:38:29.871747 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 07:38:29.871750 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 07:38:29.871753 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0506 07:38:29.871757 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0506 07:38:29.871771 1 tasks_processing.go:45] number of workers: 3 I0506 07:38:29.871790 1 tasks_processing.go:69] worker 2 listening for tasks. I0506 07:38:29.871795 1 tasks_processing.go:71] worker 2 working on remote_configuration task. I0506 07:38:29.871799 1 tasks_processing.go:69] worker 0 listening for tasks. I0506 07:38:29.871810 1 tasks_processing.go:69] worker 1 listening for tasks. I0506 07:38:29.871817 1 tasks_processing.go:71] worker 0 working on rapid_container_logs task. I0506 07:38:29.871819 1 tasks_processing.go:74] worker 1 stopped. I0506 07:38:29.871828 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0506 07:38:29.871853 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0506 07:38:29.871864 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 747ns to process 1 records I0506 07:38:29.871924 1 tasks_processing.go:74] worker 2 stopped. I0506 07:38:29.871958 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0506 07:38:29.871967 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 1.609µs to process 1 records I0506 07:38:29.872130 1 tasks_processing.go:74] worker 0 stopped. I0506 07:38:29.872147 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 303.124µs to process 0 records I0506 07:38:29.872190 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.13:53366->172.30.0.10:53: read: connection refused I0506 07:38:29.872217 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0506 07:38:29.895074 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=4e5a7aa58fabf3f00e04bdcbe82ec34058b943483082ece1d3baf1c791df1229 I0506 07:38:29.895214 1 diskrecorder.go:70] Writing 103 records to /var/lib/insights-operator/insights-2026-05-06-073829.tar.gz I0506 07:38:29.901974 1 diskrecorder.go:51] Wrote 103 records to disk in 6ms I0506 07:38:29.902015 1 periodic.go:278] Gathering cluster info every 2h0m0s I0506 07:38:29.902034 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0506 07:38:41.143683 1 configmapobserver.go:84] configmaps "insights-config" not found I0506 07:39:23.701769 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="5cfd5b7bb7847139daf65c4aea6f0b11a0c962e0f894847df81e4e1679cbc2ee") W0506 07:39:23.701807 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.crt was created I0506 07:39:23.701876 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="96217f0f2d0d27f0556614b4b821abc7c6a9a7e4369a519dfc078d8f3daf1a55") I0506 07:39:23.701876 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated" I0506 07:39:23.701916 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0506 07:39:23.701920 1 periodic.go:170] Shutting down I0506 07:39:23.701939 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0506 07:39:23.701945 1 object_count_tracker.go:151] "StorageObjectCountTracker pruner is exiting" I0506 07:39:23.701882 1 base_controller.go:181] Shutting down LoggingSyncer ... I0506 07:39:23.701960 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0506 07:39:23.701956 1 base_controller.go:181] Shutting down ConfigController ... I0506 07:39:23.701964 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0506 07:39:23.701973 1 genericapiserver.go:651] "[graceful-termination] not going to wait for active watch request(s) to drain" I0506 07:39:23.701993 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="3fafcb2b39b3cc6c9844c99cad10008de36ecb72dad3320d96912496e21c2407") I0506 07:39:23.702027 1 requestheader_controller.go:194] Shutting down RequestHeaderAuthRequestController I0506 07:39:23.702048 1 configmap_cafile_content.go:226] "Shutting down controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"