W0331 03:56:58.211265 1 cmd.go:257] Using insecure, self-signed certificates I0331 03:56:58.600553 1 start.go:138] Unable to read service ca bundle: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0331 03:56:58.601115 1 observer_polling.go:159] Starting file observer I0331 03:56:58.946894 1 operator.go:60] Starting insights-operator v0.0.0-master+$Format:%H$ I0331 03:56:58.947083 1 legacy_config.go:327] Current config: {"report":false,"storagePath":"/var/lib/insights-operator","interval":"2h","endpoint":"https://console.redhat.com/api/ingress/v1/upload","conditionalGathererEndpoint":"https://console.redhat.com/api/gathering/v2/%s/gathering_rules","pull_report":{"endpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports","delay":"60s","timeout":"3000s","min_retry":"30s"},"impersonate":"system:serviceaccount:openshift-insights:gather","enableGlobalObfuscation":false,"ocm":{"scaEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates","scaInterval":"8h","scaDisabled":false,"clusterTransferEndpoint":"https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/","clusterTransferInterval":"12h"},"disableInsightsAlerts":false,"processingStatusEndpoint":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status","reportEndpointTechPreview":"https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/report"} I0331 03:56:58.947693 1 simple_featuregate_reader.go:171] Starting feature-gate-detector I0331 03:56:58.948441 1 secure_serving.go:57] Forcing use of http/1.1 only W0331 03:56:58.948466 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected. W0331 03:56:58.948473 1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected. W0331 03:56:58.948479 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_GCM_SHA256' detected. W0331 03:56:58.948484 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_GCM_SHA384' detected. W0331 03:56:58.948489 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_128_CBC_SHA' detected. W0331 03:56:58.948493 1 secure_serving.go:69] Use of insecure cipher 'TLS_RSA_WITH_AES_256_CBC_SHA' detected. I0331 03:56:58.953840 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController I0331 03:56:58.953862 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController I0331 03:56:58.953874 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file" I0331 03:56:58.953933 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0331 03:56:58.953935 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file" I0331 03:56:58.953953 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0331 03:56:58.954022 1 operator.go:125] FeatureGates initialized: knownFeatureGates=[AdditionalRoutingCapabilities AdminNetworkPolicy AlibabaPlatform AzureWorkloadIdentity BuildCSIVolumes CPMSMachineNamePrefix ConsolePluginContentSecurityPolicy ExternalOIDC ExternalOIDCWithUIDAndExtraClaimMappings GatewayAPI GatewayAPIController HighlyAvailableArbiter ImageVolume IngressControllerLBSubnetsAWS KMSv1 MachineConfigNodes ManagedBootImages ManagedBootImagesAWS MetricsCollectionProfiles NetworkDiagnosticsConfig NetworkLiveMigration NetworkSegmentation PinnedImages ProcMountType RouteAdvertisements RouteExternalCertificate ServiceAccountTokenNodeBinding SetEIPForNLBIngressController SigstoreImageVerification StoragePerformantSecurityPolicy UpgradeStatus UserNamespacesPodSecurityStandards UserNamespacesSupport VSphereMultiDisk VSphereMultiNetworks AWSClusterHostedDNS AWSClusterHostedDNSInstall AWSDedicatedHosts AWSServiceLBNetworkSecurityGroup AutomatedEtcdBackup AzureClusterHostedDNSInstall AzureDedicatedHosts AzureMultiDisk BootImageSkewEnforcement BootcNodeManagement ClusterAPIInstall ClusterAPIInstallIBMCloud ClusterMonitoringConfig ClusterVersionOperatorConfiguration DNSNameResolver DualReplica DyanmicServiceEndpointIBMCloud DynamicResourceAllocation EtcdBackendQuota EventedPLEG Example Example2 ExternalSnapshotMetadata GCPClusterHostedDNS GCPClusterHostedDNSInstall GCPCustomAPIEndpoints GCPCustomAPIEndpointsInstall ImageModeStatusReporting ImageStreamImportMode IngressControllerDynamicConfigurationManager InsightsConfig InsightsConfigAPI InsightsOnDemandDataGather IrreconcilableMachineConfig KMSEncryptionProvider MachineAPIMigration MachineAPIOperatorDisableMachineHealthCheckController ManagedBootImagesAzure ManagedBootImagesvSphere MaxUnavailableStatefulSet MinimumKubeletVersion MixedCPUsAllocation MultiArchInstallAzure MultiDiskSetup MutatingAdmissionPolicy NewOLM NewOLMCatalogdAPIV1Metas NewOLMOwnSingleNamespace NewOLMPreflightPermissionChecks NewOLMWebhookProviderOpenshiftServiceCA NoRegistryClusterOperations NodeSwap NutanixMultiSubnets OVNObservability OpenShiftPodSecurityAdmission PreconfiguredUDNAddresses SELinuxMount ShortCertRotation SignatureStores SigstoreImageVerificationPKI TranslateStreamCloseWebsocketRequests VSphereConfigurableMaxAllowedBlockVolumesPerNode VSphereHostVMGroupZonal VSphereMixedNodeEnv VolumeAttributesClass VolumeGroupSnapshot] I0331 03:56:58.954032 1 event.go:377] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-insights", Name:"insights-operator", UID:"fa3592bb-2ea1-441c-b69b-cd0b54d8489d", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'FeatureGatesInitialized' FeatureGates updated to featuregates.Features{Enabled:[]v1.FeatureGateName{"AdditionalRoutingCapabilities", "AdminNetworkPolicy", "AlibabaPlatform", "AzureWorkloadIdentity", "BuildCSIVolumes", "CPMSMachineNamePrefix", "ConsolePluginContentSecurityPolicy", "ExternalOIDC", "ExternalOIDCWithUIDAndExtraClaimMappings", "GatewayAPI", "GatewayAPIController", "HighlyAvailableArbiter", "ImageVolume", "IngressControllerLBSubnetsAWS", "KMSv1", "MachineConfigNodes", "ManagedBootImages", "ManagedBootImagesAWS", "MetricsCollectionProfiles", "NetworkDiagnosticsConfig", "NetworkLiveMigration", "NetworkSegmentation", "PinnedImages", "ProcMountType", "RouteAdvertisements", "RouteExternalCertificate", "ServiceAccountTokenNodeBinding", "SetEIPForNLBIngressController", "SigstoreImageVerification", "StoragePerformantSecurityPolicy", "UpgradeStatus", "UserNamespacesPodSecurityStandards", "UserNamespacesSupport", "VSphereMultiDisk", "VSphereMultiNetworks"}, Disabled:[]v1.FeatureGateName{"AWSClusterHostedDNS", "AWSClusterHostedDNSInstall", "AWSDedicatedHosts", "AWSServiceLBNetworkSecurityGroup", "AutomatedEtcdBackup", "AzureClusterHostedDNSInstall", "AzureDedicatedHosts", "AzureMultiDisk", "BootImageSkewEnforcement", "BootcNodeManagement", "ClusterAPIInstall", "ClusterAPIInstallIBMCloud", "ClusterMonitoringConfig", "ClusterVersionOperatorConfiguration", "DNSNameResolver", "DualReplica", "DyanmicServiceEndpointIBMCloud", "DynamicResourceAllocation", "EtcdBackendQuota", "EventedPLEG", "Example", "Example2", "ExternalSnapshotMetadata", "GCPClusterHostedDNS", "GCPClusterHostedDNSInstall", "GCPCustomAPIEndpoints", "GCPCustomAPIEndpointsInstall", "ImageModeStatusReporting", "ImageStreamImportMode", "IngressControllerDynamicConfigurationManager", "InsightsConfig", "InsightsConfigAPI", "InsightsOnDemandDataGather", "IrreconcilableMachineConfig", "KMSEncryptionProvider", "MachineAPIMigration", "MachineAPIOperatorDisableMachineHealthCheckController", "ManagedBootImagesAzure", "ManagedBootImagesvSphere", "MaxUnavailableStatefulSet", "MinimumKubeletVersion", "MixedCPUsAllocation", "MultiArchInstallAzure", "MultiDiskSetup", "MutatingAdmissionPolicy", "NewOLM", "NewOLMCatalogdAPIV1Metas", "NewOLMOwnSingleNamespace", "NewOLMPreflightPermissionChecks", "NewOLMWebhookProviderOpenshiftServiceCA", "NoRegistryClusterOperations", "NodeSwap", "NutanixMultiSubnets", "OVNObservability", "OpenShiftPodSecurityAdmission", "PreconfiguredUDNAddresses", "SELinuxMount", "ShortCertRotation", "SignatureStores", "SigstoreImageVerificationPKI", "TranslateStreamCloseWebsocketRequests", "VSphereConfigurableMaxAllowedBlockVolumesPerNode", "VSphereHostVMGroupZonal", "VSphereMixedNodeEnv", "VolumeAttributesClass", "VolumeGroupSnapshot"}} I0331 03:56:58.954177 1 dynamic_serving_content.go:135] "Starting controller" name="serving-cert::/tmp/serving-cert-1358545322/tls.crt::/tmp/serving-cert-1358545322/tls.key" I0331 03:56:58.954671 1 secure_serving.go:213] Serving securely on [::]:8443 I0331 03:56:58.954721 1 tlsconfig.go:243] "Starting DynamicServingCertificateController" W0331 03:56:58.960161 1 configmapobserver.go:64] Cannot get the configuration config map: configmaps "insights-config" not found. Default configuration is used. I0331 03:56:58.960189 1 secretconfigobserver.go:216] Legacy configuration set: enabled=false endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=false reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0331 03:56:58.960312 1 base_controller.go:76] Waiting for caches to sync for ConfigController I0331 03:56:58.965018 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0331 03:56:58.965034 1 secretconfigobserver.go:204] Legacy configuration updated: enabled=true endpoint=https://console.redhat.com/api/ingress/v1/upload conditional_gatherer_endpoint=https://console.redhat.com/api/gathering/v2/%s/gathering_rules interval=2h0m0s token=true reportEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports initialPollingDelay=1m0s minRetryTime=30s pollingTimeout=50m0s processingStatusEndpoint=https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/request/%s/status I0331 03:56:58.969652 1 secretconfigobserver.go:119] support secret does not exist I0331 03:56:58.979769 1 secretconfigobserver.go:249] Found cloud.openshift.com token I0331 03:56:58.984301 1 secretconfigobserver.go:119] support secret does not exist I0331 03:56:58.986463 1 recorder.go:161] Pruning old reports every 7h31m27s, max age is 288h0m0s I0331 03:56:58.991345 1 periodic.go:209] Running clusterconfig gatherer I0331 03:56:58.991353 1 controllerstatus.go:80] name=insightsuploader healthy=true reason= message= I0331 03:56:58.991368 1 insightsuploader.go:86] Reporting status periodically to https://console.redhat.com/api/ingress/v1/upload every 2h0m0s, starting in 1m30s I0331 03:56:58.991394 1 tasks_processing.go:45] number of workers: 64 I0331 03:56:58.991413 1 controllerstatus.go:80] name=insightsreport healthy=true reason= message= I0331 03:56:58.991428 1 insightsreport.go:296] Starting report retriever I0331 03:56:58.991431 1 tasks_processing.go:69] worker 63 listening for tasks. I0331 03:56:58.991432 1 tasks_processing.go:69] worker 11 listening for tasks. I0331 03:56:58.991436 1 insightsreport.go:298] Insights analysis reports will be downloaded from the https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports endpoint with a delay of 1m0s I0331 03:56:58.991440 1 tasks_processing.go:71] worker 11 working on image task. I0331 03:56:58.991440 1 tasks_processing.go:69] worker 17 listening for tasks. I0331 03:56:58.991431 1 tasks_processing.go:69] worker 16 listening for tasks. I0331 03:56:58.991447 1 tasks_processing.go:69] worker 18 listening for tasks. I0331 03:56:58.991448 1 tasks_processing.go:69] worker 12 listening for tasks. I0331 03:56:58.991453 1 tasks_processing.go:69] worker 19 listening for tasks. I0331 03:56:58.991438 1 tasks_processing.go:69] worker 7 listening for tasks. I0331 03:56:58.991458 1 tasks_processing.go:69] worker 1 listening for tasks. I0331 03:56:58.991462 1 tasks_processing.go:69] worker 42 listening for tasks. I0331 03:56:58.991449 1 tasks_processing.go:69] worker 0 listening for tasks. I0331 03:56:58.991465 1 tasks_processing.go:69] worker 5 listening for tasks. I0331 03:56:58.991458 1 tasks_processing.go:69] worker 4 listening for tasks. I0331 03:56:58.991466 1 tasks_processing.go:69] worker 13 listening for tasks. I0331 03:56:58.991465 1 tasks_processing.go:69] worker 3 listening for tasks. I0331 03:56:58.991471 1 tasks_processing.go:69] worker 44 listening for tasks. I0331 03:56:58.991467 1 tasks_processing.go:69] worker 43 listening for tasks. I0331 03:56:58.991484 1 tasks_processing.go:69] worker 2 listening for tasks. I0331 03:56:58.991488 1 tasks_processing.go:69] worker 20 listening for tasks. I0331 03:56:58.991489 1 tasks_processing.go:69] worker 48 listening for tasks. I0331 03:56:58.991490 1 tasks_processing.go:69] worker 15 listening for tasks. I0331 03:56:58.991495 1 tasks_processing.go:69] worker 36 listening for tasks. I0331 03:56:58.991490 1 tasks_processing.go:69] worker 32 listening for tasks. I0331 03:56:58.991501 1 tasks_processing.go:69] worker 38 listening for tasks. I0331 03:56:58.991504 1 tasks_processing.go:69] worker 33 listening for tasks. I0331 03:56:58.991507 1 tasks_processing.go:69] worker 35 listening for tasks. I0331 03:56:58.991509 1 tasks_processing.go:69] worker 26 listening for tasks. I0331 03:56:58.991474 1 tasks_processing.go:69] worker 6 listening for tasks. I0331 03:56:58.991512 1 tasks_processing.go:69] worker 40 listening for tasks. I0331 03:56:58.991514 1 tasks_processing.go:69] worker 39 listening for tasks. I0331 03:56:58.991517 1 tasks_processing.go:69] worker 56 listening for tasks. I0331 03:56:58.991514 1 tasks_processing.go:69] worker 8 listening for tasks. I0331 03:56:58.991472 1 tasks_processing.go:69] worker 14 listening for tasks. I0331 03:56:58.991521 1 tasks_processing.go:69] worker 10 listening for tasks. I0331 03:56:58.991481 1 tasks_processing.go:69] worker 30 listening for tasks. I0331 03:56:58.991534 1 tasks_processing.go:69] worker 61 listening for tasks. I0331 03:56:58.991537 1 tasks_processing.go:69] worker 54 listening for tasks. I0331 03:56:58.991537 1 tasks_processing.go:69] worker 52 listening for tasks. I0331 03:56:58.991540 1 tasks_processing.go:69] worker 23 listening for tasks. I0331 03:56:58.991533 1 tasks_processing.go:69] worker 29 listening for tasks. I0331 03:56:58.991482 1 tasks_processing.go:69] worker 31 listening for tasks. I0331 03:56:58.991485 1 tasks_processing.go:69] worker 47 listening for tasks. I0331 03:56:58.991486 1 tasks_processing.go:69] worker 9 listening for tasks. I0331 03:56:58.991554 1 tasks_processing.go:69] worker 53 listening for tasks. I0331 03:56:58.991475 1 tasks_processing.go:69] worker 45 listening for tasks. I0331 03:56:58.991558 1 tasks_processing.go:69] worker 57 listening for tasks. I0331 03:56:58.991496 1 tasks_processing.go:69] worker 37 listening for tasks. I0331 03:56:58.991497 1 tasks_processing.go:69] worker 49 listening for tasks. I0331 03:56:58.991495 1 tasks_processing.go:69] worker 34 listening for tasks. I0331 03:56:58.991472 1 tasks_processing.go:69] worker 41 listening for tasks. I0331 03:56:58.991499 1 tasks_processing.go:69] worker 25 listening for tasks. I0331 03:56:58.991507 1 tasks_processing.go:69] worker 21 listening for tasks. I0331 03:56:58.991517 1 tasks_processing.go:69] worker 22 listening for tasks. I0331 03:56:58.991518 1 tasks_processing.go:69] worker 27 listening for tasks. I0331 03:56:58.991519 1 tasks_processing.go:69] worker 50 listening for tasks. I0331 03:56:58.991520 1 tasks_processing.go:69] worker 59 listening for tasks. I0331 03:56:58.991525 1 tasks_processing.go:69] worker 28 listening for tasks. I0331 03:56:58.991525 1 tasks_processing.go:69] worker 58 listening for tasks. I0331 03:56:58.991527 1 tasks_processing.go:69] worker 60 listening for tasks. I0331 03:56:58.991435 1 tasks_processing.go:71] worker 63 working on active_alerts task. I0331 03:56:58.991599 1 tasks_processing.go:69] worker 51 listening for tasks. I0331 03:56:58.991527 1 tasks_processing.go:69] worker 24 listening for tasks. I0331 03:56:58.991480 1 tasks_processing.go:69] worker 46 listening for tasks. I0331 03:56:58.991605 1 tasks_processing.go:69] worker 55 listening for tasks. I0331 03:56:58.991617 1 tasks_processing.go:71] worker 16 working on ingress task. I0331 03:56:58.991620 1 tasks_processing.go:71] worker 46 working on authentication task. I0331 03:56:58.991622 1 tasks_processing.go:71] worker 33 working on pod_network_connectivity_checks task. I0331 03:56:58.991623 1 tasks_processing.go:71] worker 55 working on container_runtime_configs task. I0331 03:56:58.991627 1 tasks_processing.go:71] worker 48 working on crds task. I0331 03:56:58.991630 1 tasks_processing.go:71] worker 4 working on sap_datahubs task. I0331 03:56:58.991633 1 tasks_processing.go:71] worker 36 working on validating_webhook_configurations task. I0331 03:56:58.991634 1 tasks_processing.go:71] worker 8 working on operators_pods_and_events task. I0331 03:56:58.991648 1 tasks_processing.go:71] worker 42 working on config_maps task. I0331 03:56:58.991654 1 tasks_processing.go:71] worker 18 working on machine_configs task. I0331 03:56:58.991679 1 tasks_processing.go:71] worker 35 working on openstack_dataplanenodesets task. I0331 03:56:58.991697 1 tasks_processing.go:71] worker 12 working on operators task. I0331 03:56:58.991704 1 tasks_processing.go:71] worker 32 working on service_accounts task. I0331 03:56:58.991819 1 tasks_processing.go:71] worker 1 working on ceph_cluster task. I0331 03:56:58.991949 1 tasks_processing.go:71] worker 3 working on machines task. I0331 03:56:58.991997 1 tasks_processing.go:71] worker 0 working on olm_operators task. I0331 03:56:58.992047 1 tasks_processing.go:71] worker 54 working on metrics task. I0331 03:56:58.992095 1 tasks_processing.go:71] worker 49 working on mutating_webhook_configurations task. I0331 03:56:58.992149 1 tasks_processing.go:71] worker 58 working on sap_config task. I0331 03:56:58.992180 1 tasks_processing.go:71] worker 28 working on silenced_alerts task. I0331 03:56:58.992009 1 tasks_processing.go:71] worker 13 working on openstack_version task. W0331 03:56:58.992248 1 gather_silenced_alerts.go:38] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0331 03:56:58.992259 1 tasks_processing.go:71] worker 28 working on jaegers task. I0331 03:56:58.991644 1 tasks_processing.go:71] worker 7 working on aggregated_monitoring_cr_names task. I0331 03:56:58.991631 1 tasks_processing.go:71] worker 56 working on lokistack task. I0331 03:56:58.992314 1 gather.go:177] gatherer "clusterconfig" function "silenced_alerts" took 26.556µs to process 0 records I0331 03:56:58.991627 1 tasks_processing.go:71] worker 39 working on openshift_logging task. I0331 03:56:58.991622 1 tasks_processing.go:71] worker 40 working on monitoring_persistent_volumes task. I0331 03:56:58.991616 1 tasks_processing.go:71] worker 38 working on ingress_certificates task. I0331 03:56:58.992149 1 tasks_processing.go:71] worker 44 working on machine_healthchecks task. I0331 03:56:58.992118 1 tasks_processing.go:71] worker 47 working on storage_cluster task. I0331 03:56:58.991608 1 tasks_processing.go:71] worker 17 working on cluster_apiserver task. I0331 03:56:58.991612 1 tasks_processing.go:71] worker 5 working on networks task. I0331 03:56:58.991614 1 tasks_processing.go:71] worker 2 working on infrastructures task. I0331 03:56:58.991617 1 tasks_processing.go:71] worker 14 working on overlapping_namespace_uids task. I0331 03:56:58.991617 1 tasks_processing.go:71] worker 6 working on dvo_metrics task. I0331 03:56:58.991623 1 tasks_processing.go:71] worker 20 working on cost_management_metrics_configs task. W0331 03:56:58.991623 1 gather_active_alerts.go:54] Unable to load alerts client, no alerts will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0331 03:56:58.991535 1 tasks_processing.go:69] worker 62 listening for tasks. I0331 03:56:58.992017 1 tasks_processing.go:71] worker 15 working on node_logs task. I0331 03:56:58.992994 1 tasks_processing.go:74] worker 62 stopped. I0331 03:56:58.993007 1 gather.go:177] gatherer "clusterconfig" function "active_alerts" took 1.383842ms to process 0 records I0331 03:56:58.992026 1 tasks_processing.go:71] worker 19 working on qemu_kubevirt_launcher_logs task. I0331 03:56:58.992032 1 tasks_processing.go:71] worker 37 working on schedulers task. I0331 03:56:58.992035 1 tasks_processing.go:71] worker 10 working on support_secret task. I0331 03:56:58.992039 1 tasks_processing.go:71] worker 30 working on nodenetworkstates task. I0331 03:56:58.992042 1 tasks_processing.go:71] worker 61 working on tsdb_status task. W0331 03:56:58.993363 1 gather_prometheus_tsdb_status.go:38] Unable to load metrics client, tsdb status cannot be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory W0331 03:56:58.992116 1 gather_most_recent_metrics.go:64] Unable to load metrics client, no metrics will be collected: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0331 03:56:58.992123 1 tasks_processing.go:71] worker 9 working on version task. I0331 03:56:58.993404 1 gather.go:177] gatherer "clusterconfig" function "tsdb_status" took 51.508µs to process 0 records I0331 03:56:58.993430 1 gather.go:177] gatherer "clusterconfig" function "metrics" took 1.302266ms to process 0 records I0331 03:56:58.993444 1 tasks_processing.go:74] worker 54 stopped. I0331 03:56:58.992123 1 tasks_processing.go:71] worker 52 working on feature_gates task. I0331 03:56:58.992129 1 tasks_processing.go:71] worker 53 working on install_plans task. I0331 03:56:58.992128 1 tasks_processing.go:71] worker 23 working on openshift_machine_api_events task. I0331 03:56:58.992133 1 tasks_processing.go:71] worker 45 working on machine_autoscalers task. I0331 03:56:58.992136 1 tasks_processing.go:71] worker 29 working on pdbs task. I0331 03:56:58.992140 1 tasks_processing.go:71] worker 57 working on machine_sets task. I0331 03:56:58.992141 1 tasks_processing.go:71] worker 31 working on storage_classes task. I0331 03:56:58.992145 1 tasks_processing.go:71] worker 50 working on openstack_dataplanedeployments task. I0331 03:56:58.992146 1 tasks_processing.go:71] worker 43 working on clusterroles task. I0331 03:56:58.992150 1 tasks_processing.go:71] worker 34 working on nodes task. I0331 03:56:58.992158 1 tasks_processing.go:71] worker 41 working on machine_config_pools task. I0331 03:56:58.992157 1 tasks_processing.go:71] worker 26 working on image_pruners task. I0331 03:56:58.992162 1 tasks_processing.go:71] worker 25 working on oauths task. I0331 03:56:58.992165 1 tasks_processing.go:71] worker 59 working on proxies task. I0331 03:56:58.992166 1 tasks_processing.go:71] worker 21 working on openstack_controlplanes task. I0331 03:56:58.992171 1 tasks_processing.go:71] worker 51 working on container_images task. I0331 03:56:58.992171 1 tasks_processing.go:71] worker 27 working on certificate_signing_requests task. I0331 03:56:58.992172 1 tasks_processing.go:71] worker 22 working on nodenetworkconfigurationpolicies task. I0331 03:56:58.992175 1 tasks_processing.go:71] worker 60 working on sap_pods task. I0331 03:56:58.992179 1 tasks_processing.go:71] worker 24 working on number_of_pods_and_netnamespaces_with_sdn_annotations task. I0331 03:56:58.992986 1 tasks_processing.go:71] worker 63 working on image_registries task. I0331 03:56:58.993394 1 tasks_processing.go:74] worker 61 stopped. I0331 03:56:59.014317 1 tasks_processing.go:74] worker 35 stopped. I0331 03:56:59.014332 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanenodesets" took 22.626545ms to process 0 records I0331 03:56:59.014346 1 gather.go:177] gatherer "clusterconfig" function "sap_datahubs" took 22.686361ms to process 0 records I0331 03:56:59.014354 1 tasks_processing.go:74] worker 4 stopped. I0331 03:56:59.014416 1 tasks_processing.go:74] worker 33 stopped. E0331 03:56:59.014426 1 gather.go:140] gatherer "clusterconfig" function "pod_network_connectivity_checks" failed with the error: the server could not find the requested resource (get podnetworkconnectivitychecks.controlplane.operator.openshift.io) I0331 03:56:59.014434 1 gather.go:177] gatherer "clusterconfig" function "pod_network_connectivity_checks" took 22.787462ms to process 0 records I0331 03:56:59.014456 1 tasks_processing.go:74] worker 13 stopped. I0331 03:56:59.014464 1 gather.go:177] gatherer "clusterconfig" function "openstack_version" took 22.208116ms to process 0 records I0331 03:56:59.014479 1 tasks_processing.go:74] worker 39 stopped. I0331 03:56:59.014482 1 gather.go:177] gatherer "clusterconfig" function "openshift_logging" took 22.086055ms to process 0 records I0331 03:56:59.014514 1 tasks_processing.go:74] worker 56 stopped. I0331 03:56:59.014527 1 gather.go:177] gatherer "clusterconfig" function "lokistack" took 22.184716ms to process 0 records I0331 03:56:59.014599 1 tasks_processing.go:74] worker 44 stopped. E0331 03:56:59.014611 1 gather.go:140] gatherer "clusterconfig" function "machine_healthchecks" failed with the error: machinehealthchecks.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machinehealthchecks" in API group "machine.openshift.io" at the cluster scope I0331 03:56:59.014618 1 gather.go:177] gatherer "clusterconfig" function "machine_healthchecks" took 22.101833ms to process 0 records I0331 03:56:59.014686 1 controller.go:128] Initializing last reported time to 0001-01-01T00:00:00Z I0331 03:56:59.014703 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0331 03:56:59.014709 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0331 03:56:59.014714 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0331 03:56:59.014755 1 controller.go:489] The operator is still being initialized I0331 03:56:59.014766 1 controller.go:512] The operator is healthy I0331 03:56:59.015460 1 tasks_processing.go:74] worker 46 stopped. I0331 03:56:59.015651 1 recorder.go:75] Recording config/authentication with fingerprint=f6893d9f502ce770b0f86e89b589d5d710b89e19b23f1312a016c535213396e0 I0331 03:56:59.015664 1 gather.go:177] gatherer "clusterconfig" function "authentication" took 23.826951ms to process 1 records I0331 03:56:59.017039 1 tasks_processing.go:74] worker 3 stopped. E0331 03:56:59.017052 1 gather.go:140] gatherer "clusterconfig" function "machines" failed with the error: machines.machine.openshift.io is forbidden: User "system:serviceaccount:openshift-insights:gather" cannot list resource "machines" in API group "machine.openshift.io" at the cluster scope I0331 03:56:59.017063 1 gather.go:177] gatherer "clusterconfig" function "machines" took 25.071547ms to process 0 records I0331 03:56:59.017148 1 tasks_processing.go:74] worker 55 stopped. I0331 03:56:59.017159 1 gather.go:177] gatherer "clusterconfig" function "container_runtime_configs" took 25.510576ms to process 0 records I0331 03:56:59.017320 1 tasks_processing.go:74] worker 0 stopped. I0331 03:56:59.017340 1 gather.go:177] gatherer "clusterconfig" function "olm_operators" took 25.267417ms to process 0 records I0331 03:56:59.017347 1 gather.go:177] gatherer "clusterconfig" function "storage_cluster" took 24.732311ms to process 0 records I0331 03:56:59.017351 1 tasks_processing.go:74] worker 47 stopped. I0331 03:56:59.017366 1 tasks_processing.go:74] worker 28 stopped. I0331 03:56:59.017392 1 gather.go:177] gatherer "clusterconfig" function "jaegers" took 25.099862ms to process 0 records I0331 03:56:59.017398 1 gather.go:177] gatherer "clusterconfig" function "sap_config" took 25.206229ms to process 0 records I0331 03:56:59.017406 1 tasks_processing.go:74] worker 58 stopped. I0331 03:56:59.017597 1 tasks_processing.go:74] worker 11 stopped. I0331 03:56:59.017704 1 recorder.go:75] Recording config/image with fingerprint=5c50fc3ae48b5a0ab71ad32c856e3dcb1703af3c7263e3b1bc84d72d1192cd48 I0331 03:56:59.017721 1 gather.go:177] gatherer "clusterconfig" function "image" took 26.144273ms to process 1 records I0331 03:56:59.017855 1 tasks_processing.go:74] worker 16 stopped. I0331 03:56:59.017897 1 recorder.go:75] Recording config/ingress with fingerprint=f5841e26e5c57e0aaa80ea854136a89a13be8e3c67030df4b8811a5b64a8b88e I0331 03:56:59.017909 1 gather.go:177] gatherer "clusterconfig" function "ingress" took 26.006438ms to process 1 records I0331 03:56:59.018003 1 tasks_processing.go:74] worker 17 stopped. I0331 03:56:59.018072 1 recorder.go:75] Recording config/apiserver with fingerprint=a72d1f62928c6025f57b56641acaa24c86f23fa2b73c0907606d94507e04c493 I0331 03:56:59.018083 1 gather.go:177] gatherer "clusterconfig" function "cluster_apiserver" took 25.03789ms to process 1 records I0331 03:56:59.018092 1 gather.go:177] gatherer "clusterconfig" function "cost_management_metrics_configs" took 24.940739ms to process 0 records I0331 03:56:59.018105 1 tasks_processing.go:74] worker 20 stopped. I0331 03:56:59.018202 1 tasks_processing.go:74] worker 5 stopped. I0331 03:56:59.018257 1 recorder.go:75] Recording config/network with fingerprint=ecbaed9eead916b12bd888fae1904b13b87d356aec102f6407496eeedb86ae09 I0331 03:56:59.018269 1 gather.go:177] gatherer "clusterconfig" function "networks" took 25.314197ms to process 1 records I0331 03:56:59.018277 1 gather.go:177] gatherer "clusterconfig" function "node_logs" took 25.114133ms to process 0 records I0331 03:56:59.018283 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkstates" took 24.946617ms to process 0 records I0331 03:56:59.018291 1 tasks_processing.go:74] worker 30 stopped. I0331 03:56:59.018295 1 tasks_processing.go:74] worker 15 stopped. I0331 03:56:59.018328 1 tasks_processing.go:74] worker 10 stopped. E0331 03:56:59.018346 1 gather.go:140] gatherer "clusterconfig" function "support_secret" failed with the error: secrets "support" not found I0331 03:56:59.018358 1 gather.go:177] gatherer "clusterconfig" function "support_secret" took 25.101376ms to process 0 records I0331 03:56:59.018502 1 tasks_processing.go:74] worker 57 stopped. I0331 03:56:59.018518 1 gather.go:177] gatherer "clusterconfig" function "machine_sets" took 24.826627ms to process 0 records I0331 03:56:59.018633 1 tasks_processing.go:74] worker 23 stopped. I0331 03:56:59.018653 1 gather.go:177] gatherer "clusterconfig" function "openshift_machine_api_events" took 25.059417ms to process 0 records I0331 03:56:59.018672 1 gather.go:177] gatherer "clusterconfig" function "monitoring_persistent_volumes" took 26.159377ms to process 0 records I0331 03:56:59.018682 1 tasks_processing.go:74] worker 40 stopped. I0331 03:56:59.018692 1 tasks_processing.go:74] worker 49 stopped. I0331 03:56:59.018957 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/aws-pod-identity with fingerprint=1c4c78ebeedc7db1df6acfe3781d847ac7f9305d07f8a4128a58214e9e641c82 I0331 03:56:59.019009 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-podimagespec-mutation with fingerprint=3202b3602f3c2b30d953bf6e99e92a3648ccae30f4267a664a9634da46692472 I0331 03:56:59.019058 1 recorder.go:75] Recording config/mutatingwebhookconfigurations/sre-service-mutation with fingerprint=57c247cf54fa8f35b19557c448350adfea11e696f370aec226f3273ff911459f I0331 03:56:59.019071 1 gather.go:177] gatherer "clusterconfig" function "mutating_webhook_configurations" took 26.537432ms to process 3 records I0331 03:56:59.019156 1 recorder.go:75] Recording config/pdbs/openshift-image-registry/image-registry with fingerprint=c92031f48930558a2d3b8574b3bdf9923006860686186cf6091b3e148911c385 I0331 03:56:59.019186 1 recorder.go:75] Recording config/pdbs/openshift-ingress/router-default with fingerprint=3a3babf1ac7057fac11ad6fa32f01ec318763b0bc2ba075ab19fd82f3a82b298 I0331 03:56:59.019204 1 recorder.go:75] Recording config/pdbs/openshift-operator-lifecycle-manager/packageserver-pdb with fingerprint=722b14ee98169d3163fb08636e29b70e04500492616cb271d885280ed5d0343b I0331 03:56:59.019214 1 gather.go:177] gatherer "clusterconfig" function "pdbs" took 25.108977ms to process 3 records I0331 03:56:59.019158 1 tasks_processing.go:74] worker 29 stopped. I0331 03:56:59.019299 1 tasks_processing.go:74] worker 26 stopped. I0331 03:56:59.019343 1 gather_logs.go:145] no pods in namespace were found I0331 03:56:59.019452 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/imagepruner/cluster with fingerprint=1f5fb3ff87ac15b7f8adc2bf9efa1715b0ff6011d040b2b4c26a4f993b241d6f I0331 03:56:59.019464 1 gather.go:177] gatherer "clusterconfig" function "image_pruners" took 24.771051ms to process 1 records I0331 03:56:59.019473 1 gather.go:177] gatherer "clusterconfig" function "machine_autoscalers" took 25.2976ms to process 0 records I0331 03:56:59.019479 1 gather.go:177] gatherer "clusterconfig" function "openstack_dataplanedeployments" took 25.033753ms to process 0 records I0331 03:56:59.019485 1 gather.go:177] gatherer "clusterconfig" function "ceph_cluster" took 27.165028ms to process 0 records I0331 03:56:59.019489 1 gather.go:177] gatherer "clusterconfig" function "nodenetworkconfigurationpolicies" took 24.59549ms to process 0 records I0331 03:56:59.019498 1 tasks_processing.go:74] worker 45 stopped. I0331 03:56:59.019506 1 tasks_processing.go:74] worker 22 stopped. I0331 03:56:59.019508 1 tasks_processing.go:74] worker 1 stopped. I0331 03:56:59.019512 1 tasks_processing.go:74] worker 50 stopped. I0331 03:56:59.019542 1 recorder.go:75] Recording config/proxy with fingerprint=1b90e0c2a54c8bec6fbffbce496eee2699328d643f1989830ffc35026bece717 I0331 03:56:59.019555 1 gather.go:177] gatherer "clusterconfig" function "proxies" took 24.917106ms to process 1 records I0331 03:56:59.019561 1 gather.go:177] gatherer "clusterconfig" function "openstack_controlplanes" took 24.850997ms to process 0 records I0331 03:56:59.019564 1 tasks_processing.go:74] worker 59 stopped. I0331 03:56:59.019583 1 tasks_processing.go:74] worker 21 stopped. I0331 03:56:59.019649 1 tasks_processing.go:74] worker 2 stopped. I0331 03:56:59.020264 1 recorder.go:75] Recording config/infrastructure with fingerprint=e1024f8d2dfff0c1beaf12ac88614cf3165e6154e8599ddf2af8120ac1d05151 I0331 03:56:59.020281 1 gather.go:177] gatherer "clusterconfig" function "infrastructures" took 26.553444ms to process 1 records I0331 03:56:59.020290 1 gather.go:177] gatherer "clusterconfig" function "qemu_kubevirt_launcher_logs" took 26.266167ms to process 0 records I0331 03:56:59.020339 1 tasks_processing.go:74] worker 19 stopped. I0331 03:56:59.020398 1 tasks_processing.go:74] worker 25 stopped. I0331 03:56:59.020747 1 recorder.go:75] Recording config/oauth with fingerprint=ae3da669768ab1a08488766fd2c8b4399738f62fa36411ee80cbcd6b0dfa3295 I0331 03:56:59.020768 1 gather.go:177] gatherer "clusterconfig" function "oauths" took 25.410072ms to process 1 records I0331 03:56:59.020812 1 recorder.go:75] Recording config/schedulers/cluster with fingerprint=5909d64e83f06389ee6d51e0c9de16006968402a9b130e22f8dab93e35a57424 I0331 03:56:59.020819 1 gather.go:177] gatherer "clusterconfig" function "schedulers" took 26.483736ms to process 1 records I0331 03:56:59.020823 1 gather.go:177] gatherer "clusterconfig" function "sap_pods" took 25.29715ms to process 0 records I0331 03:56:59.020830 1 tasks_processing.go:74] worker 37 stopped. I0331 03:56:59.020850 1 tasks_processing.go:74] worker 60 stopped. I0331 03:56:59.020920 1 tasks_processing.go:74] worker 52 stopped. I0331 03:56:59.021316 1 recorder.go:75] Recording config/featuregate with fingerprint=43f0e5cdaa1d84d33553aee223023562bbe267c34699df109eb786947e844a2c I0331 03:56:59.021393 1 gather.go:177] gatherer "clusterconfig" function "feature_gates" took 26.710953ms to process 1 records I0331 03:56:59.021747 1 tasks_processing.go:74] worker 34 stopped. I0331 03:56:59.022403 1 recorder.go:75] Recording config/node/ip-10-0-2-201.ec2.internal with fingerprint=35d723185a8a9b6c2f9ab7227ba3173d51d0405e6765fd0268d69e004ba8a971 I0331 03:56:59.022458 1 gather.go:177] gatherer "clusterconfig" function "nodes" took 26.634683ms to process 1 records I0331 03:56:59.022657 1 tasks_processing.go:74] worker 36 stopped. I0331 03:56:59.022701 1 recorder.go:75] Recording config/validatingwebhookconfigurations/multus.openshift.io with fingerprint=699c317c0d4d41ddffa68b4b23b71686d16ad1074396f9ca8c940ef19655c7dc I0331 03:56:59.022905 1 recorder.go:75] Recording config/validatingwebhookconfigurations/network-node-identity.openshift.io with fingerprint=a3ee2789d508368091db0d82ba734cf6d29079265778065b13bbfe840282b68b I0331 03:56:59.022961 1 recorder.go:75] Recording config/validatingwebhookconfigurations/performance-addon-operator with fingerprint=e08246fc950d7e567fb64577a320ef23194d5d565617c3772dbb7f4e3e1675ce I0331 03:56:59.023388 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterrolebindings-validation with fingerprint=52ef18f2c552ce9b07c64c188dbb6bf8658dd1821b339ab88e36b57ec4229b72 I0331 03:56:59.023451 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-clusterroles-validation with fingerprint=2e9642c42081d935c6f88bd8b5efd53250a49b319c197831cd3de1b9de641ff5 I0331 03:56:59.023496 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-ingress-config-validation with fingerprint=7697af7c2bf788611f38d53365c1cff641648aab103bc11ea9e49d56517a196e I0331 03:56:59.023538 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-network-operator-validation with fingerprint=fe6b94a2a893ace402dc0e47aaec5bcffe3e85ab96ae5076e08ad37d01ac0e96 I0331 03:56:59.023604 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-regular-user-validation with fingerprint=4cd2dc89e9563aa6ccc33d0e1bc011983c812875501748a9697b31c77bf10f46 I0331 03:56:59.023645 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-scc-validation with fingerprint=09a40cc06b5558c06c22280f7dc61371e58d55560e945c356a8b9e792354ba00 I0331 03:56:59.023670 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-serviceaccount-validation with fingerprint=02670ef06a6d3c2646d8a2d22e5fbb119be41d989a8bf781ab862b8825b836e4 I0331 03:56:59.023693 1 recorder.go:75] Recording config/validatingwebhookconfigurations/sre-techpreviewnoupgrade-validation with fingerprint=7169f9720197d6251f31fbc719ba8fc8157355c70ebb299fff72d3c030e65570 I0331 03:56:59.023711 1 gather.go:177] gatherer "clusterconfig" function "validating_webhook_configurations" took 30.597068ms to process 11 records I0331 03:56:59.023808 1 recorder.go:75] Recording config/storage/storageclasses/gp2-csi with fingerprint=6103bdad8921073c3b2d04995e6167f5696a74ad01b56238f8a86d48196f1734 I0331 03:56:59.023814 1 tasks_processing.go:74] worker 31 stopped. I0331 03:56:59.023823 1 recorder.go:75] Recording config/storage/storageclasses/gp3-csi with fingerprint=c60b9bbe13aaf6aef42ca3ae4eded96702e211f3300f9a382aef2f464d377fd9 I0331 03:56:59.023829 1 gather.go:177] gatherer "clusterconfig" function "storage_classes" took 29.183908ms to process 2 records I0331 03:56:59.024129 1 tasks_processing.go:74] worker 14 stopped. I0331 03:56:59.024157 1 recorder.go:75] Recording config/namespaces_with_overlapping_uids with fingerprint=4f53cda18c2baa0c0354bb5f9a3ecbe5ed12ab4d8e11ba873c2f11161202b945 I0331 03:56:59.024171 1 gather.go:177] gatherer "clusterconfig" function "overlapping_namespace_uids" took 31.264488ms to process 1 records W0331 03:56:59.024264 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0331 03:56:59.026095 1 tasks_processing.go:74] worker 63 stopped. I0331 03:56:59.026649 1 recorder.go:75] Recording config/clusteroperator/imageregistry.operator.openshift.io/config/cluster with fingerprint=269d1b8eb15092645125422ac867611a1c84231185de25bb724e4eb0c3a72593 I0331 03:56:59.026665 1 gather.go:177] gatherer "clusterconfig" function "image_registries" took 31.524627ms to process 1 records I0331 03:56:59.026678 1 gather.go:177] gatherer "clusterconfig" function "certificate_signing_requests" took 31.846279ms to process 0 records I0331 03:56:59.026687 1 tasks_processing.go:74] worker 27 stopped. I0331 03:56:59.030880 1 tasks_processing.go:74] worker 41 stopped. I0331 03:56:59.030896 1 gather.go:177] gatherer "clusterconfig" function "machine_config_pools" took 36.896552ms to process 0 records I0331 03:56:59.032921 1 tasks_processing.go:74] worker 48 stopped. I0331 03:56:59.034438 1 recorder.go:75] Recording config/crd/volumesnapshots.snapshot.storage.k8s.io with fingerprint=e3fa0c65533bbbef8b8ea1f4299c00096332153b25f9c44002fb4a09aaf89bad I0331 03:56:59.034930 1 recorder.go:75] Recording config/crd/volumesnapshotcontents.snapshot.storage.k8s.io with fingerprint=7e4d8c0e34dc2207f878b4d0a4c0279ece7b05fa4dc0adf9b5f143045b3be139 I0331 03:56:59.034981 1 gather.go:177] gatherer "clusterconfig" function "crds" took 41.27793ms to process 2 records I0331 03:56:59.035017 1 gather.go:177] gatherer "clusterconfig" function "aggregated_monitoring_cr_names" took 40.924897ms to process 0 records I0331 03:56:59.035174 1 sca.go:136] Pulling SCA certificates from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates. Next check is in 8h0m0s W0331 03:56:59.034938 1 operator.go:288] started I0331 03:56:59.034963 1 base_controller.go:76] Waiting for caches to sync for LoggingSyncer I0331 03:56:59.035285 1 cluster_transfer.go:83] checking the availability of cluster transfer. Next check is in 12h0m0s I0331 03:56:59.035318 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/admin with fingerprint=e125863c64d14806e4100f8b5cd9e7b74a6315c7ceeeeed94a3065a0f7307fab I0331 03:56:59.035328 1 tasks_processing.go:74] worker 43 stopped. I0331 03:56:59.035335 1 tasks_processing.go:74] worker 7 stopped. I0331 03:56:59.035769 1 recorder.go:75] Recording cluster-scoped-resources/rbac.authorization.k8s.io/clusterroles/edit with fingerprint=10554c0a094e1dc015e37b710f5d34f8d156a63def3b701cee2adae7b067c020 I0331 03:56:59.035791 1 gather.go:177] gatherer "clusterconfig" function "clusterroles" took 40.365454ms to process 2 records I0331 03:56:59.038601 1 tasks_processing.go:74] worker 51 stopped. I0331 03:56:59.038697 1 recorder.go:75] Recording config/running_containers with fingerprint=9ae8f2e3a0291ca0606d1f8d09643d2d4082fa3b403d87c28ec1c1671ba83178 I0331 03:56:59.038719 1 gather.go:177] gatherer "clusterconfig" function "container_images" took 44.237544ms to process 1 records I0331 03:56:59.042784 1 tasks_processing.go:74] worker 24 stopped. I0331 03:56:59.042797 1 gather.go:177] gatherer "clusterconfig" function "number_of_pods_and_netnamespaces_with_sdn_annotations" took 48.232322ms to process 0 records I0331 03:56:59.054091 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController I0331 03:56:59.054106 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file I0331 03:56:59.054131 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file I0331 03:56:59.057980 1 prometheus_rules.go:88] Prometheus rules successfully created I0331 03:56:59.059833 1 controller.go:212] Source scaController *sca.Controller is not ready I0331 03:56:59.059844 1 controller.go:212] Source clusterTransferController *clustertransfer.Controller is not ready I0331 03:56:59.059848 1 controller.go:212] Source periodic-clusterconfig *controllerstatus.Simple is not ready I0331 03:56:59.059850 1 controller.go:212] Source periodic-conditional *controllerstatus.Simple is not ready I0331 03:56:59.059852 1 controller.go:212] Source periodic-workloads *controllerstatus.Simple is not ready I0331 03:56:59.059868 1 controller.go:489] The operator is still being initialized I0331 03:56:59.059873 1 controller.go:512] The operator is healthy I0331 03:56:59.060710 1 base_controller.go:82] Caches are synced for ConfigController I0331 03:56:59.060720 1 base_controller.go:119] Starting #1 worker of ConfigController controller ... E0331 03:56:59.067236 1 cluster_transfer.go:95] failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%2720273850-a2b6-4bdc-a2bd-d7bf471f4f58%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:45113->172.30.0.10:53: read: connection refused I0331 03:56:59.067248 1 controllerstatus.go:80] name=clusterTransferController healthy=true reason=Disconnected message=failed to pull cluster transfer: unable to retrieve cluster transfer data from https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/: Get "https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/?search=cluster_uuid+is+%2720273850-a2b6-4bdc-a2bd-d7bf471f4f58%27+and+status+is+%27accepted%27": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:45113->172.30.0.10:53: read: connection refused I0331 03:56:59.075841 1 tasks_processing.go:74] worker 42 stopped. E0331 03:56:59.075858 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "cluster-monitoring-config" not found E0331 03:56:59.075869 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "gateway-mode-config" not found E0331 03:56:59.075874 1 gather.go:140] gatherer "clusterconfig" function "config_maps" failed with the error: configmaps "insights-config" not found I0331 03:56:59.075887 1 recorder.go:75] Recording config/configmaps/openshift-config/installer-images/images.json with fingerprint=26b6661162b099a0f5a279859b4f46c867929a79d9a4a41fde4be4e6fe138018 I0331 03:56:59.075923 1 recorder.go:75] Recording config/configmaps/openshift-config/kube-root-ca.crt/ca.crt with fingerprint=d476c7d3f5b104863f08f481b1264dcc68cc272ecefb0ecb709b18a6afab034d I0331 03:56:59.075934 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/invoker with fingerprint=76b482f683cd3ef9da02debac5b26080a5aeb06ff768ee5c21117514dff29d8a I0331 03:56:59.075940 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-install/version with fingerprint=c93090eb0d2a4736885abeb79c91680cfd01fda46464f83456b085d4dc8239f0 I0331 03:56:59.075946 1 recorder.go:75] Recording config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 I0331 03:56:59.076017 1 recorder.go:75] Recording config/configmaps/openshift-config/rosa-brand-logo/rosa-brand-logo.svg with fingerprint=6ed8ca4dd7a8eee7249182bc006e9649ce84d76c551ddfaaa33e55d8c4cc1ed0 I0331 03:56:59.076028 1 recorder.go:75] Recording config/configmaps/kube-system/cluster-config-v1/install-config with fingerprint=ab3811c6b83fd7b8e920094cfa3080d1b4ee3c35ec4c8379437b21d27bd6608d I0331 03:56:59.076037 1 gather.go:177] gatherer "clusterconfig" function "config_maps" took 84.171253ms to process 7 records I0331 03:56:59.076176 1 configmapobserver.go:84] configmaps "insights-config" not found I0331 03:56:59.110576 1 tasks_processing.go:74] worker 9 stopped. I0331 03:56:59.110860 1 recorder.go:75] Recording config/version with fingerprint=f91620c7131171d3258b02420fe843f1746c8fd02e5ab39571357a541445271a I0331 03:56:59.110873 1 recorder.go:75] Recording config/id with fingerprint=9244ad45c719bb5bffd76fc90c63e7136d330b086b0af5ef6f58039e8e7829db I0331 03:56:59.110883 1 gather.go:177] gatherer "clusterconfig" function "version" took 117.168563ms to process 2 records I0331 03:56:59.113873 1 tasks_processing.go:74] worker 18 stopped. I0331 03:56:59.113895 1 recorder.go:75] Recording aggregated/unused_machine_configs_count with fingerprint=4bfc9fa984e5dfcd45848faaf05269de7619bf42edf9f781751af5ee05c1a499 I0331 03:56:59.113903 1 gather.go:177] gatherer "clusterconfig" function "machine_configs" took 122.200418ms to process 1 records I0331 03:56:59.120170 1 requests.go:205] Asking for SCA certificate with "{"arch": ["x86_64"]}" payload W0331 03:56:59.123577 1 sca.go:161] Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:42972->172.30.0.10:53: read: connection refused I0331 03:56:59.123589 1 controllerstatus.go:80] name=scaController healthy=true reason=NonHTTPError message=Failed to pull SCA certs from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: unable to retrieve SCA certs data from https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates: Post "https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates": dial tcp: lookup api.openshift.com on 172.30.0.10:53: read udp 10.128.0.21:42972->172.30.0.10:53: read: connection refused I0331 03:56:59.134776 1 tasks_processing.go:74] worker 38 stopped. E0331 03:56:59.134788 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret 'router-certs-default' in namespace 'openshift-ingress': secrets "router-certs-default" not found E0331 03:56:59.134794 1 gather.go:140] gatherer "clusterconfig" function "ingress_certificates" failed with the error: failed to get secret '2pcf4goiisa5m55tcrhfco9v7iv9c0o4-primary-cert-bundle-secret' in namespace 'openshift-ingress-operator': secrets "2pcf4goiisa5m55tcrhfco9v7iv9c0o4-primary-cert-bundle-secret" not found I0331 03:56:59.134842 1 recorder.go:75] Recording aggregated/ingress_controllers_certs with fingerprint=52a053f8bc7075ff999eac79b6d898b5d1f40f8c1bf67150f08fd9d45efa85a2 I0331 03:56:59.134853 1 gather.go:177] gatherer "clusterconfig" function "ingress_certificates" took 142.291381ms to process 1 records I0331 03:56:59.135808 1 base_controller.go:82] Caches are synced for LoggingSyncer I0331 03:56:59.135819 1 base_controller.go:119] Starting #1 worker of LoggingSyncer controller ... I0331 03:56:59.432979 1 gather_cluster_operator_pods_and_events.go:121] Found 10 pods with 11 containers I0331 03:56:59.432991 1 gather_cluster_operator_pods_and_events.go:235] Maximum buffer size: 2287802 bytes I0331 03:56:59.433017 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns container dns-default-j5vbk pod in namespace openshift-dns (previous: false). I0331 03:56:59.655913 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-j5vbk pod in namespace openshift-dns for failing operator dns (previous: false): "container \"dns\" in pod \"dns-default-j5vbk\" is waiting to start: ContainerCreating" I0331 03:56:59.655930 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"dns\" in pod \"dns-default-j5vbk\" is waiting to start: ContainerCreating" I0331 03:56:59.655938 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for kube-rbac-proxy container dns-default-j5vbk pod in namespace openshift-dns (previous: false). I0331 03:56:59.838856 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for dns-default-j5vbk pod in namespace openshift-dns for failing operator kube-rbac-proxy (previous: false): "container \"kube-rbac-proxy\" in pod \"dns-default-j5vbk\" is waiting to start: ContainerCreating" I0331 03:56:59.838875 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"kube-rbac-proxy\" in pod \"dns-default-j5vbk\" is waiting to start: ContainerCreating" I0331 03:56:59.838886 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for dns-node-resolver container node-resolver-cpvvz pod in namespace openshift-dns (previous: false). W0331 03:57:00.021601 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0331 03:57:00.035805 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0331 03:57:00.035855 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-668f7887c4-k69cr pod in namespace openshift-image-registry (previous: false). I0331 03:57:00.230233 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0331 03:57:00.230286 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-668f7887c4-q8js8 pod in namespace openshift-image-registry (previous: false). I0331 03:57:00.438316 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for image-registry-668f7887c4-q8js8 pod in namespace openshift-image-registry for failing operator registry (previous: false): "container \"registry\" in pod \"image-registry-668f7887c4-q8js8\" is waiting to start: ContainerCreating" I0331 03:57:00.438332 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"registry\" in pod \"image-registry-668f7887c4-q8js8\" is waiting to start: ContainerCreating" I0331 03:57:00.438367 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for registry container image-registry-9569fc645-kl4j7 pod in namespace openshift-image-registry (previous: false). I0331 03:57:00.452369 1 gather_cluster_operators.go:184] Unable to get configs.samples.operator.openshift.io resource due to: configs.samples.operator.openshift.io "cluster" not found I0331 03:57:00.633795 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0331 03:57:00.633810 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for node-ca container node-ca-lxz6d pod in namespace openshift-image-registry (previous: false). I0331 03:57:00.836388 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0331 03:57:00.836402 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-678d4d8689-dhfqd pod in namespace openshift-ingress (previous: false). I0331 03:57:00.852974 1 tasks_processing.go:74] worker 12 stopped. I0331 03:57:00.853016 1 recorder.go:75] Recording config/clusteroperator/console with fingerprint=2307ec1d54b8a31b19ef09081e90b780c4d360eb26d249d162922a4309e5cfd6 I0331 03:57:00.853042 1 recorder.go:75] Recording config/clusteroperator/csi-snapshot-controller with fingerprint=d4fdc2a4393ce64edd5b0a629b0785bcd2d380f214b96b15870a138e5665307d I0331 03:57:00.853071 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/csisnapshotcontroller/cluster with fingerprint=5adc514f4b63e2f1ecc68bf6f9c0af70c5eea04522a49524e102721b1c41f80e I0331 03:57:00.853095 1 recorder.go:75] Recording config/clusteroperator/dns with fingerprint=7d578edb24b34ae48ad69e97c0273d581245a9fac4b0f318ea1971a5e4486554 I0331 03:57:00.853110 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/dns/default with fingerprint=9e7b4ce029030d3d8c3b49af92c556acdcc415000b40d3f969dbdc42c432b47f I0331 03:57:00.853144 1 recorder.go:75] Recording config/clusteroperator/image-registry with fingerprint=d322112612898e2841bf63fb4f3483a376b4d63800b07936c914dc0a138011e6 I0331 03:57:00.853177 1 recorder.go:75] Recording config/clusteroperator/ingress with fingerprint=d7d9a107c46924f792cbe04d418db957a07fbd697f7da6ab9052e4d95773a9df I0331 03:57:00.853203 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/ingresscontroller/openshift-ingress-operator/default with fingerprint=4d61f395a705f849ebe31efd4f2f737ed1a315d4102bf9440200795e2d57c385 I0331 03:57:00.853232 1 recorder.go:75] Recording config/clusteroperator/insights with fingerprint=31a880979af004f82a4e9142e4722fdc71cf1d280c715cac029dcb97d4336883 I0331 03:57:00.853251 1 recorder.go:75] Recording config/clusteroperator/kube-apiserver with fingerprint=ad7a5ae04bc4f8e0eec3ae8dbed4e754b9c2dd7e27c76e6f077e2350ed8db6e3 I0331 03:57:00.853260 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubeapiserver/cluster with fingerprint=51503bf0b784fcf65ea46bcaf1f72ac1a5c4d5dc211934f18f27871efed05762 I0331 03:57:00.853284 1 recorder.go:75] Recording config/clusteroperator/kube-controller-manager with fingerprint=13f3a28bf6957bde3edba4cef997196160bf08e2a66cad4bc69e8b9740ff4752 I0331 03:57:00.853300 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubecontrollermanager/cluster with fingerprint=ce90c0d4f367d7da085074268031798382ae7c54fdcb0a21f15a4818fe308c11 I0331 03:57:00.853317 1 recorder.go:75] Recording config/clusteroperator/kube-scheduler with fingerprint=59070e5b30ecc3a5dd8206e8e95e26685fe2be64c5f00d8ce18b115a3840e0ec I0331 03:57:00.853327 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubescheduler/cluster with fingerprint=f2940fb9fd20c19951dfc295eb363b7fba0c505f5ae61f01967a063099e6b60a I0331 03:57:00.853340 1 recorder.go:75] Recording config/clusteroperator/kube-storage-version-migrator with fingerprint=537ea14afe5b77b2685d3e0b82fc4cb4174844b20f9d569e05e6f665ce97c6e5 I0331 03:57:00.853360 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/kubestorageversionmigrator/cluster with fingerprint=9351181aa7e6ada41ef581ab31e13516c6b934cc95710154bafb2eb222cb58db I0331 03:57:00.853395 1 recorder.go:75] Recording config/clusteroperator/monitoring with fingerprint=b899e5b8cc3011ea95730145d902939ec3c380d62e93d792f1db56c2c18480ab I0331 03:57:00.853512 1 recorder.go:75] Recording config/clusteroperator/network with fingerprint=4059abce14724c5ce8e1217b9aeadb4cf0d24a01388d451eedd82947dd2ab585 I0331 03:57:00.853521 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/ovn with fingerprint=626a89d20e0deaed5b6dfb533acfe65f4bb1618bd200a703b62e60c5d16d94ab I0331 03:57:00.853527 1 recorder.go:75] Recording config/clusteroperator/network.operator.openshift.io/operatorpki/openshift-ovn-kubernetes/signer with fingerprint=90410b16914712b85b3c4578716ad8c0ae072e688f4cd1e022bf76f20da3506d I0331 03:57:00.853548 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/network/cluster with fingerprint=a93d15eaecb455a0e40ecb2826eeecc1533899204ddd3c3921d15ab70af7ae75 I0331 03:57:00.853570 1 recorder.go:75] Recording config/clusteroperator/node-tuning with fingerprint=3e9703a1bc15f34d0889de70f218faf6ee5cf0aaa8e4342c17020728a49c79e6 I0331 03:57:00.853598 1 recorder.go:75] Recording config/clusteroperator/openshift-apiserver with fingerprint=776f934bded04aeaa535f2b03a435c31f2a1bbe6dc6dc2352a1b2cdeefacc18f I0331 03:57:00.853607 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftapiserver/cluster with fingerprint=e712e6cf27339b441e4ed1f4cde91dbde7e952698ba93407e4457db63a4a4c76 I0331 03:57:00.853624 1 recorder.go:75] Recording config/clusteroperator/openshift-controller-manager with fingerprint=3d091d93fc024e959b487aabca4f69beadc966e610f6d8c2919d85057ac103f2 I0331 03:57:00.853630 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/openshiftcontrollermanager/cluster with fingerprint=d71a0f4672f9b45d9fc8293bf1687afc650fd28d32e2e30de27523fe7b4eadf7 I0331 03:57:00.853644 1 recorder.go:75] Recording config/clusteroperator/openshift-samples with fingerprint=ae326fdc1825b764d14abfddb01fde408d100b9e1f9c267ead7337864264d912 I0331 03:57:00.853657 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager with fingerprint=92328d8231f25710698192f70a455b4c584233a0b3979bacadbc3717a579beda I0331 03:57:00.853670 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-catalog with fingerprint=97d1d1bd4d9258caeffe219b41211d58ab38cd9b5d6dc0082413967d3b1c55ad I0331 03:57:00.853684 1 recorder.go:75] Recording config/clusteroperator/operator-lifecycle-manager-packageserver with fingerprint=85b89a38b0731aab89559115368196a5009e19e35433c6ddd460503df065ed21 I0331 03:57:00.853704 1 recorder.go:75] Recording config/clusteroperator/service-ca with fingerprint=eeb69be31f2b07e68fdcff2484c280438ae094c4cf0daf75f8fee6a908d1e511 I0331 03:57:00.853741 1 recorder.go:75] Recording config/clusteroperator/storage with fingerprint=8bc6678d591181dae6f0a7ec3becc1eccd4b66cd9129b100d5fa389df4c92d8b I0331 03:57:00.853758 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/clustercsidriver/ebs.csi.aws.com with fingerprint=510064d6f6bcced87ab5bd2ddaff3d0edd7f93f4a4f7af2641f29fc53ffab21e I0331 03:57:00.853767 1 recorder.go:75] Recording config/clusteroperator/operator.openshift.io/storage/cluster with fingerprint=8e480f8c1ce1b39baac42d8ec780c57c2592929ae0c801b61ffad49ba13f33ad I0331 03:57:00.853774 1 gather.go:177] gatherer "clusterconfig" function "operators" took 1.861259992s to process 35 records W0331 03:57:01.022614 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. I0331 03:57:01.030490 1 gather_cluster_operator_pods_and_events.go:280] Error: "log buffer is empty" I0331 03:57:01.030504 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-678d4d8689-fw4zd pod in namespace openshift-ingress (previous: false). I0331 03:57:01.239167 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-678d4d8689-fw4zd pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-678d4d8689-fw4zd\" is waiting to start: ContainerCreating" I0331 03:57:01.239183 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-678d4d8689-fw4zd\" is waiting to start: ContainerCreating" I0331 03:57:01.239194 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for router container router-default-f89c8d44f-zvt4j pod in namespace openshift-ingress (previous: false). I0331 03:57:01.435549 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for router-default-f89c8d44f-zvt4j pod in namespace openshift-ingress for failing operator router (previous: false): "container \"router\" in pod \"router-default-f89c8d44f-zvt4j\" is waiting to start: ContainerCreating" I0331 03:57:01.435567 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"router\" in pod \"router-default-f89c8d44f-zvt4j\" is waiting to start: ContainerCreating" I0331 03:57:01.435578 1 gather_cluster_operator_pods_and_events.go:365] Fetching logs for serve-healthcheck-canary container ingress-canary-gnprj pod in namespace openshift-ingress-canary (previous: false). I0331 03:57:01.635350 1 gather_cluster_operator_pods_and_events.go:408] Failed to fetch log for ingress-canary-gnprj pod in namespace openshift-ingress-canary for failing operator serve-healthcheck-canary (previous: false): "container \"serve-healthcheck-canary\" in pod \"ingress-canary-gnprj\" is waiting to start: ContainerCreating" I0331 03:57:01.635371 1 gather_cluster_operator_pods_and_events.go:280] Error: "container \"serve-healthcheck-canary\" in pod \"ingress-canary-gnprj\" is waiting to start: ContainerCreating" I0331 03:57:01.635405 1 tasks_processing.go:74] worker 8 stopped. I0331 03:57:01.635488 1 recorder.go:75] Recording events/openshift-dns with fingerprint=c7312fe73c92c434bed57a8dbda6418d6e33b1050832ab012a49167f895e9ff5 I0331 03:57:01.635566 1 recorder.go:75] Recording events/openshift-image-registry with fingerprint=fc122487e921686d117a88efd620d69b1c4aa40146b290bf3969873bb50c3b9c I0331 03:57:01.635593 1 recorder.go:75] Recording events/openshift-ingress-operator with fingerprint=037e11cc5d11a8f3bbeaf6932fa0887f1d1092f5eff5e0cf08594390d1e0c865 I0331 03:57:01.635636 1 recorder.go:75] Recording events/openshift-ingress with fingerprint=62845d9205f03464ad97b277d97bcea48da3fa353e282d04cb92e3a6ec52ff43 I0331 03:57:01.635647 1 recorder.go:75] Recording events/openshift-ingress-canary with fingerprint=233003722de36cedc80f5e45413c5f09ff279b007f3750e99119d562ab4fba4f I0331 03:57:01.636963 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-668f7887c4-k69cr with fingerprint=fb8a40000a6de0cd016c0382ff8b0170fde2b31593178dce14812de5fe20b298 I0331 03:57:01.637076 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-668f7887c4-q8js8 with fingerprint=df7257cd575a9edf0a823ddb3359ab36867d7e8ce8ba090316f6a1a8c551ee81 I0331 03:57:01.637151 1 recorder.go:75] Recording config/pod/openshift-image-registry/image-registry-9569fc645-kl4j7 with fingerprint=37aa6e441ce5594bd303575bbd4385020c4b2e276fa3e7abe5421b13420fcfa8 I0331 03:57:01.637160 1 gather.go:177] gatherer "clusterconfig" function "operators_pods_and_events" took 2.643757093s to process 8 records W0331 03:57:02.022228 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0331 03:57:03.021711 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0331 03:57:04.021409 1 gather_dvo_metrics.go:210] Failed to read the DVO metrics. Trying again. W0331 03:57:04.021432 1 gather_dvo_metrics.go:117] Unable to read metrics from endpoint "http://deployment-validation-operator-metrics.openshift-deployment-validation-operator.svc:8383": DVO metrics service was not available within the 5s timeout: context deadline exceeded I0331 03:57:04.021443 1 tasks_processing.go:74] worker 6 stopped. E0331 03:57:04.021452 1 gather.go:140] gatherer "clusterconfig" function "dvo_metrics" failed with the error: DVO metrics service was not available within the 5s timeout: context deadline exceeded I0331 03:57:04.021462 1 recorder.go:75] Recording config/dvo_metrics with fingerprint=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 W0331 03:57:04.021475 1 gather.go:155] issue recording gatherer "clusterconfig" function "dvo_metrics" result "config/dvo_metrics" because of the warning: warning: the record with the same fingerprint "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855" was already recorded at path "config/configmaps/openshift-config/openshift-service-ca.crt/service-ca.crt", recording another one with a different path "config/dvo_metrics" I0331 03:57:04.021484 1 gather.go:177] gatherer "clusterconfig" function "dvo_metrics" took 5.028506029s to process 1 records I0331 03:57:11.431143 1 tasks_processing.go:74] worker 53 stopped. I0331 03:57:11.431187 1 recorder.go:75] Recording config/installplans with fingerprint=f17dbfacc3bfddf27ca3b213b39495434cd4c4e9e3dbd69566ffb3845bbcf539 I0331 03:57:11.431200 1 gather.go:177] gatherer "clusterconfig" function "install_plans" took 12.43760701s to process 1 records I0331 03:57:12.125991 1 configmapobserver.go:84] configmaps "insights-config" not found I0331 03:57:12.200286 1 tasks_processing.go:74] worker 32 stopped. I0331 03:57:12.200563 1 recorder.go:75] Recording config/serviceaccounts with fingerprint=751aa8b7d1208af5fd734152a53c8f2354708612b0cd50cfd9c595651a5cc8c8 I0331 03:57:12.200579 1 gather.go:177] gatherer "clusterconfig" function "service_accounts" took 13.208563124s to process 1 records E0331 03:57:12.200642 1 periodic.go:247] "Unhandled Error" err="clusterconfig failed after 13.209s with: function \"pod_network_connectivity_checks\" failed with an error, function \"machine_healthchecks\" failed with an error, function \"machines\" failed with an error, function \"support_secret\" failed with an error, function \"config_maps\" failed with an error, function \"ingress_certificates\" failed with an error, function \"dvo_metrics\" failed with an error" I0331 03:57:12.201773 1 controllerstatus.go:89] name=periodic-clusterconfig healthy=false reason=PeriodicGatherFailed message=Source clusterconfig could not be retrieved: function "pod_network_connectivity_checks" failed with an error, function "machine_healthchecks" failed with an error, function "machines" failed with an error, function "support_secret" failed with an error, function "config_maps" failed with an error, function "ingress_certificates" failed with an error, function "dvo_metrics" failed with an error I0331 03:57:12.201792 1 periodic.go:209] Running workloads gatherer I0331 03:57:12.201812 1 tasks_processing.go:45] number of workers: 2 I0331 03:57:12.201831 1 tasks_processing.go:69] worker 1 listening for tasks. I0331 03:57:12.201839 1 tasks_processing.go:71] worker 1 working on workload_info task. I0331 03:57:12.201843 1 tasks_processing.go:69] worker 0 listening for tasks. I0331 03:57:12.201853 1 tasks_processing.go:71] worker 0 working on helmchart_info task. I0331 03:57:12.224369 1 gather_workloads_info.go:278] Loaded pods in 0s, will wait 19s for image data I0331 03:57:12.233010 1 tasks_processing.go:74] worker 0 stopped. I0331 03:57:12.233026 1 gather.go:177] gatherer "workloads" function "helmchart_info" took 31.142188ms to process 0 records I0331 03:57:12.236843 1 gather_workloads_info.go:387] No image sha256:765f0d23b637f685f98a31bd47c131b03cf72a40761a3f9a9d6320faa3c33733 (13ms) I0331 03:57:12.246458 1 gather_workloads_info.go:387] No image sha256:2904a78e2eb73fd6a9bb94c105c2a056831fb4113fbb7b0607c50adc9d879c9b (10ms) I0331 03:57:12.255207 1 gather_workloads_info.go:387] No image sha256:5a95c19d82767e0235b4edb4a0536482c816904897aae1dc3eb255cb52b87a9f (9ms) I0331 03:57:12.265934 1 gather_workloads_info.go:387] No image sha256:2e57e192c3c1240fd935dcd55c8fde5e70e78bf81d6176c96edf21fafe59f8ba (11ms) I0331 03:57:12.275490 1 gather_workloads_info.go:387] No image sha256:289816958633a763a72dbc44e1dad40466223164e7e253039514f0d974ea5d21 (10ms) I0331 03:57:12.288228 1 gather_workloads_info.go:387] No image sha256:695cf2f0cc07683c2a3ce1eaf3e56fe18abc6e2bac716f7d9843f5d173b9df52 (13ms) I0331 03:57:12.298480 1 gather_workloads_info.go:387] No image sha256:5f0b67cfbbc381243fb91ccc17345b56d05f4d717c667e8c644e5bf05633ba71 (10ms) I0331 03:57:12.308173 1 gather_workloads_info.go:387] No image sha256:7adc1eab05d6724c76ba751f6df816b08d6e70b78dee9eb94fa6fd9690542c98 (10ms) I0331 03:57:12.318245 1 gather_workloads_info.go:387] No image sha256:521712486e2c6e3c020dad6a1cb340db8e55665b69f7c208fab9cd9e965fd588 (10ms) I0331 03:57:12.327040 1 gather_workloads_info.go:387] No image sha256:943018739e3db1763c3184b460dbc409e058abbac76d57b9927faad317be85e4 (9ms) I0331 03:57:12.337255 1 gather_workloads_info.go:387] No image sha256:7b31223098f08328f5ddea8e5b871dbbd5f5a61ec550e8956f66793c0c6031a9 (10ms) I0331 03:57:12.435942 1 gather_workloads_info.go:387] No image sha256:a56211d075aa43cbb491f669a5b2e46ee023dc95b7d51dbac28f463948c5ad61 (99ms) I0331 03:57:12.535202 1 gather_workloads_info.go:387] No image sha256:4556896f77307821531ef91b7b7faccb82b824ea695693b2989f597f0deca038 (99ms) I0331 03:57:12.636206 1 gather_workloads_info.go:387] No image sha256:e84cb128d930bd1ab867cc89b7b7bf2b2c0e41105ab93b5381069945b3ee9c57 (101ms) I0331 03:57:12.734636 1 gather_workloads_info.go:387] No image sha256:2e564f336c77116053f34d4201d364d8da04e789cfffa0ea422574c95f2d6404 (98ms) I0331 03:57:12.836198 1 gather_workloads_info.go:387] No image sha256:a258c226562adb14e3a163a1940938526ee6a0928982a7667d85d9a7334ce639 (102ms) I0331 03:57:12.935418 1 gather_workloads_info.go:387] No image sha256:a498046d64605bcccee2440aa4f04a4602baaae263cf01d977ec5208e876b1fd (99ms) I0331 03:57:13.036874 1 gather_workloads_info.go:387] No image sha256:ca1344cb64140188b7cae7bbc51fb751566c0b0c97d5e39b5850e628032c4a5e (101ms) I0331 03:57:13.134142 1 gather_workloads_info.go:387] No image sha256:653c666f842c13e0baae2e89a9b1efe0e2ef56f621ffb5b32005115d2a26ab8c (97ms) I0331 03:57:13.235898 1 gather_workloads_info.go:387] No image sha256:36b9e89c3cfcf1ab9ae500486e38afb6862cba48cb0b4d84a09508ab8f3d299f (102ms) I0331 03:57:13.338224 1 gather_workloads_info.go:387] No image sha256:03cf4cd7ef1518610c6c7b3ad27d1622d82e98e3dc6e3f8e5d0fceb5c8d3786e (102ms) I0331 03:57:13.437026 1 gather_workloads_info.go:387] No image sha256:a0105d1eb62cf6ac9e5e2ef28d3e89bf6dc514bc594fc7090fe5a5ee18a09c87 (99ms) I0331 03:57:13.536320 1 gather_workloads_info.go:387] No image sha256:b3909bf664c77097f75b3768830863d642eed3815dab2bfb4415c771ca2d5007 (99ms) I0331 03:57:13.634970 1 gather_workloads_info.go:387] No image sha256:2598489729a4b258e4ecda4a06f6875133f2a10ced5c5241f8a57a8a05418e36 (99ms) I0331 03:57:13.736186 1 gather_workloads_info.go:387] No image sha256:56a85660a445eced5c79a595a0eccf590087c5672d50f49d4c25ad52f9a44f04 (101ms) I0331 03:57:13.835969 1 gather_workloads_info.go:387] No image sha256:91828234f107c068c8a4966d08370ae7b73e637651dbc6d92c18c4553402c22c (100ms) I0331 03:57:13.839006 1 configmapobserver.go:84] configmaps "insights-config" not found I0331 03:57:13.934672 1 gather_workloads_info.go:387] No image sha256:0a99240166165eb5718e7516a43282fe32df9c7c5e809b31b58abe44e42ff94d (99ms) I0331 03:57:14.036482 1 gather_workloads_info.go:387] No image sha256:47154813651033d59751fb655a384dbffb64dd26f10bd7f3be0c3128d0486356 (102ms) I0331 03:57:14.040109 1 configmapobserver.go:84] configmaps "insights-config" not found I0331 03:57:14.135884 1 gather_workloads_info.go:387] No image sha256:c940ea87e7d133d75ba0002ef00c0806825eed3db8094cdb260d1bac18127733 (99ms) I0331 03:57:14.237988 1 gather_workloads_info.go:387] No image sha256:c15ca0c0ad60fe8757c2d5d1723fcdd7a1ed6c0251a90d22a7e6cae6811d01aa (102ms) I0331 03:57:14.336164 1 gather_workloads_info.go:387] No image sha256:1a2532940843248c57d52141185dd71fbc393ab28b65d48f682038632c1dbbad (98ms) I0331 03:57:14.336190 1 tasks_processing.go:74] worker 1 stopped. E0331 03:57:14.336201 1 gather.go:140] gatherer "workloads" function "workload_info" failed with the error: no running pods found for the insights-runtime-extractor statefulset I0331 03:57:14.336495 1 recorder.go:75] Recording config/workload_info with fingerprint=66a5f2b36eb3afc926c830cc98a4346abdf4d94eae4d324198e8c1f5706c6989 I0331 03:57:14.336510 1 gather.go:177] gatherer "workloads" function "workload_info" took 2.134343795s to process 1 records E0331 03:57:14.336534 1 periodic.go:247] "Unhandled Error" err="workloads failed after 2.134s with: function \"workload_info\" failed with an error" I0331 03:57:14.337626 1 controllerstatus.go:89] name=periodic-workloads healthy=false reason=PeriodicGatherFailed message=Source workloads could not be retrieved: function "workload_info" failed with an error I0331 03:57:14.337637 1 periodic.go:209] Running conditional gatherer I0331 03:57:14.343544 1 requests.go:294] Making HTTP GET request at: https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules I0331 03:57:14.349738 1 conditional_gatherer.go:107] Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.21:33806->172.30.0.10:53: read: connection refused E0331 03:57:14.350071 1 conditional_gatherer.go:322] unable to update alerts cache: open /var/run/configmaps/service-ca-bundle/service-ca.crt: no such file or directory I0331 03:57:14.350139 1 conditional_gatherer.go:384] updating version cache for conditional gatherer I0331 03:57:14.356496 1 conditional_gatherer.go:392] cluster version is '4.20.8' E0331 03:57:14.356509 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:57:14.356514 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:57:14.356518 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:57:14.356521 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:57:14.356524 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:57:14.356527 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:57:14.356530 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:57:14.356533 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing E0331 03:57:14.356538 1 conditional_gatherer.go:209] error checking conditions for a gathering rule: alerts cache is missing I0331 03:57:14.356555 1 tasks_processing.go:45] number of workers: 3 I0331 03:57:14.356561 1 tasks_processing.go:69] worker 2 listening for tasks. I0331 03:57:14.356564 1 tasks_processing.go:71] worker 2 working on conditional_gatherer_rules task. I0331 03:57:14.356568 1 tasks_processing.go:69] worker 0 listening for tasks. I0331 03:57:14.356577 1 tasks_processing.go:71] worker 0 working on remote_configuration task. I0331 03:57:14.356580 1 tasks_processing.go:69] worker 1 listening for tasks. I0331 03:57:14.356589 1 tasks_processing.go:74] worker 1 stopped. I0331 03:57:14.356591 1 tasks_processing.go:71] worker 2 working on rapid_container_logs task. I0331 03:57:14.356655 1 recorder.go:75] Recording insights-operator/conditional-gatherer-rules with fingerprint=7034af97e7e41c22e4b775abdd4b9066c8ebb19da33eb7f69f39bfd2eb5f6406 I0331 03:57:14.356668 1 gather.go:177] gatherer "conditional" function "conditional_gatherer_rules" took 858ns to process 1 records I0331 03:57:14.356709 1 recorder.go:75] Recording insights-operator/remote-configuration with fingerprint=0394430c431eec4d48bb1811a90918e95161d2282c59af26f2473613cc0959db I0331 03:57:14.356717 1 gather.go:177] gatherer "conditional" function "remote_configuration" took 1.207µs to process 1 records I0331 03:57:14.356728 1 tasks_processing.go:74] worker 0 stopped. I0331 03:57:14.356833 1 tasks_processing.go:74] worker 2 stopped. I0331 03:57:14.356845 1 gather.go:177] gatherer "conditional" function "rapid_container_logs" took 226.428µs to process 0 records I0331 03:57:14.356864 1 controllerstatus.go:89] name=periodic-conditional healthy=false reason=NotAvailable message=Get "https://console.redhat.com/api/gathering/v2/4.20.8/gathering_rules": dial tcp: lookup console.redhat.com on 172.30.0.10:53: read udp 10.128.0.21:33806->172.30.0.10:53: read: connection refused I0331 03:57:14.356881 1 recorder.go:75] Recording insights-operator/remote-configuration.json with fingerprint=359de9c990c741675cec72fda96b5c3682221efdb4799f5eaa6e9805bcd3b5c1 I0331 03:57:14.380240 1 recorder.go:75] Recording insights-operator/gathers with fingerprint=a977e51fe7f3fe6b9c3ee12dc478ad87954fbd5138106ffb8917bc9effb074f9 I0331 03:57:14.380397 1 diskrecorder.go:70] Writing 99 records to /var/lib/insights-operator/insights-2026-03-31-035714.tar.gz I0331 03:57:14.389041 1 diskrecorder.go:51] Wrote 99 records to disk in 8ms I0331 03:57:14.389073 1 periodic.go:278] Gathering cluster info every 2h0m0s I0331 03:57:14.389095 1 periodic.go:279] Configuration is dataReporting: interval: 2h0m0s, uploadEndpoint: https://console.redhat.com/api/ingress/v1/upload, storagePath: /var/lib/insights-operator, downloadEndpoint: https://console.redhat.com/api/insights-results-aggregator/v2/cluster/%s/reports, conditionalGathererEndpoint: https://console.redhat.com/api/gathering/v2/%s/gathering_rules, obfuscation: [] sca: disabled: false, endpoint: https://api.openshift.com/api/accounts_mgmt/v1/entitlement_certificates, interval: 8h0m0s alerting: disabled: false clusterTransfer: endpoint: https://api.openshift.com/api/accounts_mgmt/v1/cluster_transfers/, interval: 12h0m0s proxy: httpProxy: , httpsProxy: , noProxy: I0331 03:57:29.011713 1 configmapobserver.go:84] configmaps "insights-config" not found I0331 03:58:13.602242 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.key" has been created (hash="6b1df72cc6ec939200d4537852fa93562984d7e69ac7a2a5508b832b68be6f75") W0331 03:58:13.602272 1 builder.go:160] Restart triggered because of file /var/run/secrets/serving-cert/tls.key was created I0331 03:58:13.602321 1 simple_featuregate_reader.go:177] Shutting down feature-gate-detector I0331 03:58:13.602349 1 base_controller.go:181] Shutting down ConfigController ... I0331 03:58:13.602352 1 observer_polling.go:111] Observed file "/var/run/configmaps/service-ca-bundle/service-ca.crt" has been created (hash="14288d35abe72befcb8f644ae72becc941a2487a785c8df260c1daad7379803a") I0331 03:58:13.602367 1 periodic.go:170] Shutting down I0331 03:58:13.602366 1 genericapiserver.go:693] "[graceful-termination] pre-shutdown hooks completed" name="PreShutdownHooksStopped" I0331 03:58:13.602410 1 observer_polling.go:111] Observed file "/var/run/secrets/serving-cert/tls.crt" has been created (hash="830538c7097ba19751dbc0ebcc08376f10aec4fe9f8ee213f276e1584169aebc") I0331 03:58:13.602418 1 base_controller.go:181] Shutting down LoggingSyncer ... I0331 03:58:13.602432 1 base_controller.go:123] Shutting down worker of LoggingSyncer controller ... I0331 03:58:13.602440 1 base_controller.go:113] All LoggingSyncer workers have been terminated I0331 03:58:13.602434 1 base_controller.go:123] Shutting down worker of ConfigController controller ... I0331 03:58:13.602448 1 genericapiserver.go:548] "[graceful-termination] shutdown event" name="ShutdownInitiated"