==================== ISSUES SUMMARY ==================== ========== ODH/RHOAI ISSUES SUMMARY (status, events, highlights only) ========== 158m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 122m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 122m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 53m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 112s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 89s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 122m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 122m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning FailedScheduling pod/mlflow-mg-3120-g1-8vmxx running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "mlflow-mg-3120-g1-8vmxx" not found 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 89s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 112s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 53m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 158m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning ClusterIPNotAllocated service/mlflow Cluster IP [IPv4]: 172.30.254.145 is not allocated; repairing 158m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 158m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning FailedToRetrieveImagePullSecret pod/mlflow-mg-3120-g1-7qj5q Unable to retrieve some image pull secrets (mlflow-sa-dockercfg-xvf64); attempting to pull the image may not succeed. 40m Warning FailedMount pod/mlflow-mg-3120-g1-7qj5q MountVolume.SetUp failed for volume "mlflow-tls" : secret "mlflow-tls" not found 28m Warning Deleting modelregistry/postgres-registry-1783232526563 Custom Resource postgres-registry-1783232526563 is being deleted from the namespace rhoai-model-registries 25m Warning Deleting modelregistry/default-db-registry-1783232526563 Custom Resource default-db-registry-1783232526563 is being deleted from the namespace rhoai-model-registries 28m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 73m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 27m Warning Deleting modelregistry/default-db-registry-1783232526563 Custom Resource default-db-registry-1783232526563 is being deleted from the namespace rhoai-model-registries 27m Warning Deleting modelregistry/default-db-registry-1783232526563 Custom Resource default-db-registry-1783232526563 is being deleted from the namespace rhoai-model-registries 25m Warning Deleting modelregistry/default-db-registry-1783232526563 Custom Resource default-db-registry-1783232526563 is being deleted from the namespace rhoai-model-registries 23m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 23m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 74m Warning Unhealthy pod/db-model-registry0-865b4f5d6f-k924m Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 73m Warning FailedMount pod/model-registry0-7bb5b65f65-wrf47 MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "model-registry0-kube-rbac-proxy" not found 73m Warning FailedMount pod/model-registry0-7bb5b65f65-wrf47 MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "model-registry0-kube-rbac-proxy" not found 29m Warning Unhealthy pod/mysql-db-registry-1783232526563-b6695fdfb-tbcxz Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 27m Warning Unhealthy pod/default-db-registry-1783232526563-65c44894ff-gclcz Startup probe failed: Get "http://10.130.0.160:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 27m Warning Unhealthy pod/default-db-registry-1783232526563-65c44894ff-gclcz Startup probe failed: HTTP probe failed with statuscode: 503 26m Warning Unhealthy pod/default-db-registry-1783232526563-postgres-748f9c885-p6q24 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 26m Warning Unhealthy pod/default-db-registry-1783232526563-65c44894ff-fghr4 Startup probe failed: Get "http://10.131.0.176:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/default-db-registry-1783232526563-65c44894ff-fghr4 Startup probe failed: HTTP probe failed with statuscode: 503 25m Warning Unhealthy pod/mysql-db-registry-1783232526563-b6695fdfb-tbcxz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 25m Warning Unhealthy pod/default-db-registry-1783232526563-postgres-748f9c885-hwvk9 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 25m Warning Unhealthy pod/model-registry-db-662874-864c6b9cc8-sx6hq Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 23m Warning Unhealthy pod/model-registry-db-662874-864c6b9cc8-sx6hq Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 {"level":"info","ts":"2026-07-05T06:46:28Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"error occurred during rule evaluation: clusterrole.rbac.authorization.k8s.io \"kueue-batch-user-role\" not found","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c"} --------------------------------------------------------------------------------------------------- ==================== STATUS REPORT ==================== ========== ODH/RHOAI STATUS REPORT (2026-07-05T06:52:23Z) ========== --- OpenShift / OCP version --- Client Version: 4.20.8 Kustomize Version: v5.6.0 Server Version: 4.21.19 Kubernetes Version: v1.34.8 --- Namespaces --- namespace/redhat-ods-applications namespace/redhat-ods-monitoring namespace/redhat-ods-operator namespace/rhoai-model-registries namespace/rhods-notebooks --- Subscriptions (redhat-ods-operator) --- NAME PACKAGE SOURCE CHANNEL rhods-operator rhods-operator rhoai-catalog-dev beta Subscription conditions (unhealthy): CatalogSourcesUnhealthy: status=False reason=AllCatalogSourcesHealthy message=all available catalogsources are healthy --- ClusterServiceVersions (redhat-ods-operator) --- NAME DISPLAY VERSION REPLACES PHASE authorino-operator.v1.4.1 Authorino Operator 1.4.1 authorino-operator.v1.4.0 Succeeded cert-manager-operator.v1.20.0 cert-manager Operator for Red Hat OpenShift 1.20.0 cert-manager-operator.v1.19.0 Succeeded custom-metrics-autoscaler.v2.19.0-1 Custom Metrics Autoscaler 2.19.0-1 custom-metrics-autoscaler.v2.18.1-2 Succeeded dns-operator.v1.4.0 DNS Operator 1.4.0 dns-operator.v1.3.1 Succeeded kueue-operator.v1.3.1 Red Hat build of Kueue 1.3.1 Succeeded limitador-operator.v1.4.0 Limitador Operator 1.4.0 limitador-operator.v1.3.1 Succeeded rhcl-operator.v1.4.1 Red Hat Connectivity Link 1.4.1 rhcl-operator.v1.4.0 Succeeded rhods-operator.3.5.0-ea.1 Red Hat OpenShift AI 3.5.0-ea.1 Succeeded servicemeshoperator3.v3.3.5 Red Hat OpenShift Service Mesh 3 3.3.5 servicemeshoperator3.v3.2.0 Succeeded --- CSVs not in Succeeded phase --- (all CSVs report phase Succeeded) --- InstallPlans (redhat-ods-operator, not Complete) --- (no non-Complete InstallPlans) --- DataScienceCluster --- NAME READY REASON default-dsc True DSC Ready summary: True reason= DSC component states: aipipelines: managementState=Managed dashboard: managementState=Managed feastoperator: managementState=Managed kserve: managementState=Managed kueue: managementState=Unmanaged llamastackoperator: managementState=n/a mlflowoperator: managementState=Managed modelregistry: managementState=Managed ogx: managementState=Managed ray: managementState=Managed sparkoperator: managementState=Managed trainer: managementState=Managed trainingoperator: managementState=Managed trustyai: managementState=Managed workbenches: managementState=Managed DSC conditions: Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= ComponentsReady: status=True reason= message= AIPipelinesReady: status=True reason= message= DashboardReady: status=True reason= message= FeastOperatorReady: status=True reason= message= KserveLLMInferenceServiceDependencies: status=True reason= message= KserveLLMInferenceServiceWideEPDependencies: status=True reason= message= KserveReady: status=True reason= message= KueueReady: status=True reason= message= MLflowOperatorReady: status=True reason= message= ModelControllerReady: status=True reason= message= ModelRegistryReady: status=True reason= message= ModelsAsServiceReady: status=True reason=Reconciled message=MaaS platform manifests applied and maas-api deployment is available OGXReady: status=True reason= message= RayReady: status=True reason= message= SparkOperatorReady: status=True reason= message= TrainerReady: status=True reason= message= TrainingOperatorReady: status=True reason= message= TrustyAIReady: status=True reason= message= WorkbenchesReady: status=True reason= message= --- DSCInitialization --- NAME AGE PHASE CREATED AT default-dsci 3d15h Ready 2026-07-01T15:48:58Z DSCInitialization conditions: ReconcileComplete: status=True reason=ReconcileCompleted message=Reconcile completed successfully Available: status=True reason=ReconcileCompleted message=Reconcile completed successfully Progressing: status=False reason=ReconcileCompleted message=Reconcile completed successfully Degraded: status=False reason=ReconcileCompleted message=Reconcile completed successfully Upgradeable: status=True reason=ReconcileCompleted message=Reconcile completed successfully MonitoringReady: status=True reason=Ready message=Monitoring stack is initialized Ready: status=True reason= message= ProvisioningSucceeded: status=True reason= message= MonitoringStackAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR TempoAvailable: status=False reason=TracesNotConfigured message=Traces not configured in DSCI CR OpenTelemetryCollectorAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Metrics not configured in DSCI CR Traces not configured in DSCI CR AlertingAvailable: status=False reason=AlertingNotConfigured message=Alerting not configured in DSCI CR ThanosQuerierAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR PersesAvailable: status=False reason=MetricsNotConfiguredAndTracesNotConfigured message=Perses requires at least Metrics or Traces to be configured NodeMetricsEndpointAvailable: status=False reason=MetricsNotConfigured message=Metrics not configured in DSCI CR --- Component CRs (Ready) --- NAME READY REASON URL dashboard.components.platform.opendatahub.io/default-dashboard True NAME READY REASON datasciencepipelines.components.platform.opendatahub.io/default-datasciencepipelines True NAME READY REASON feastoperator.components.platform.opendatahub.io/default-feastoperator True NAME READY REASON kserve.components.platform.opendatahub.io/default-kserve True NAME READY REASON kueue.components.platform.opendatahub.io/default-kueue True NAME READY REASON mlflowoperator.components.platform.opendatahub.io/default-mlflowoperator True NAME READY REASON URI modelcontroller.components.platform.opendatahub.io/default-modelcontroller True NAME READY REASON modelregistry.components.platform.opendatahub.io/default-modelregistry True NAME READY REASON modelsasservice.components.platform.opendatahub.io/default-modelsasservice True NAME READY REASON ogx.components.platform.opendatahub.io/default-ogx True NAME READY REASON ray.components.platform.opendatahub.io/default-ray True NAME READY REASON sparkoperator.components.platform.opendatahub.io/default-sparkoperator True NAME READY REASON trainer.components.platform.opendatahub.io/default-trainer True NAME READY REASON trainingoperator.components.platform.opendatahub.io/default-trainingoperator True NAME READY REASON trustyai.components.platform.opendatahub.io/default-trustyai True NAME READY REASON workbenches.components.platform.opendatahub.io/default-workbenches True --- Deployments not fully ready (ODH/RHOAI namespaces) --- (all deployments report ready replicas == desired) --- Pods not Running/Succeeded (ODH/RHOAI namespaces) --- (all pods Running or Succeeded) --------------------------------------------------------------------------------------------------- ==================== EVENTS ==================== ========== ODH/RHOAI EVENTS (Warning / Error / Failed) ========== --- namespace/redhat-ods-applications --- LAST SEEN TYPE REASON OBJECT MESSAGE 158m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 122m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 122m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 53m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 112s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 89s Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 122m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 122m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning FailedScheduling pod/mlflow-mg-3120-g1-8vmxx running PreBind plugin "VolumeBinding": binding volumes: pod does not exist any more: pod "mlflow-mg-3120-g1-8vmxx" not found 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 89s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 92s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 112s Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 14m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 145m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 37m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 50m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 53m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 60m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 63m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 65m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 66m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 69m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 70m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 71m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 158m Warning InvalidConfiguration servicemonitor/controller-manager-metrics-monitor "redhat-ods-applications/controller-manager-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 73m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 86m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 109m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 46m Warning ClusterIPNotAllocated service/mlflow Cluster IP [IPv4]: 172.30.254.145 is not allocated; repairing 158m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 158m Warning InvalidConfiguration servicemonitor/odh-model-controller-metrics-monitor "redhat-ods-applications/odh-model-controller-metrics-monitor" was rejected due to invalid configuration: endpoints[0]: it accesses file system via bearer token file which Prometheus specification prohibits 40m Warning FailedToRetrieveImagePullSecret pod/mlflow-mg-3120-g1-7qj5q Unable to retrieve some image pull secrets (mlflow-sa-dockercfg-xvf64); attempting to pull the image may not succeed. 40m Warning FailedMount pod/mlflow-mg-3120-g1-7qj5q MountVolume.SetUp failed for volume "mlflow-tls" : secret "mlflow-tls" not found --- namespace/redhat-ods-monitoring --- (no events) --- namespace/redhat-ods-operator --- (no events) --- namespace/rhoai-model-registries --- LAST SEEN TYPE REASON OBJECT MESSAGE 28m Warning Deleting modelregistry/postgres-registry-1783232526563 Custom Resource postgres-registry-1783232526563 is being deleted from the namespace rhoai-model-registries 25m Warning Deleting modelregistry/default-db-registry-1783232526563 Custom Resource default-db-registry-1783232526563 is being deleted from the namespace rhoai-model-registries 28m Warning Deleting modelregistry/e2e-test-registry Custom Resource e2e-test-registry is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 74m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 73m Warning Deleting modelregistry/model-registry0 Custom Resource model-registry0 is being deleted from the namespace rhoai-model-registries 27m Warning Deleting modelregistry/default-db-registry-1783232526563 Custom Resource default-db-registry-1783232526563 is being deleted from the namespace rhoai-model-registries 27m Warning Deleting modelregistry/default-db-registry-1783232526563 Custom Resource default-db-registry-1783232526563 is being deleted from the namespace rhoai-model-registries 25m Warning Deleting modelregistry/default-db-registry-1783232526563 Custom Resource default-db-registry-1783232526563 is being deleted from the namespace rhoai-model-registries 23m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 23m Warning Deleting modelregistry/test-model-registry-662874 Custom Resource test-model-registry-662874 is being deleted from the namespace rhoai-model-registries 74m Warning Unhealthy pod/db-model-registry0-865b4f5d6f-k924m Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 73m Warning FailedMount pod/model-registry0-7bb5b65f65-wrf47 MountVolume.SetUp failed for volume "kube-rbac-proxy-key" : secret "model-registry0-kube-rbac-proxy" not found 73m Warning FailedMount pod/model-registry0-7bb5b65f65-wrf47 MountVolume.SetUp failed for volume "kube-rbac-proxy-cert" : secret "model-registry0-kube-rbac-proxy" not found 29m Warning Unhealthy pod/mysql-db-registry-1783232526563-b6695fdfb-tbcxz Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 27m Warning Unhealthy pod/default-db-registry-1783232526563-65c44894ff-gclcz Startup probe failed: Get "http://10.130.0.160:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 27m Warning Unhealthy pod/default-db-registry-1783232526563-65c44894ff-gclcz Startup probe failed: HTTP probe failed with statuscode: 503 26m Warning Unhealthy pod/default-db-registry-1783232526563-postgres-748f9c885-p6q24 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 26m Warning Unhealthy pod/default-db-registry-1783232526563-65c44894ff-fghr4 Startup probe failed: Get "http://10.131.0.176:8080/readyz/isDirty": context deadline exceeded (Client.Timeout exceeded while awaiting headers) 26m Warning Unhealthy pod/default-db-registry-1783232526563-65c44894ff-fghr4 Startup probe failed: HTTP probe failed with statuscode: 503 25m Warning Unhealthy pod/mysql-db-registry-1783232526563-b6695fdfb-tbcxz Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 25m Warning Unhealthy pod/default-db-registry-1783232526563-postgres-748f9c885-hwvk9 Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 25m Warning Unhealthy pod/model-registry-db-662874-864c6b9cc8-sx6hq Readiness probe failed: mysql: [Warning] Using a password on the command line interface can be insecure.... 23m Warning Unhealthy pod/model-registry-db-662874-864c6b9cc8-sx6hq Readiness probe errored and resulted in unknown state: rpc error: code = Unknown desc = command error: cannot register an exec PID: container is stopping, stdout: , stderr: , exit code -1 --- namespace/rhods-notebooks --- LAST SEEN TYPE REASON OBJECT MESSAGE --------------------------------------------------------------------------------------------------- ==================== OPERATOR LOG HIGHLIGHTS ==================== ========== ODH/RHOAI LOG HIGHLIGHTS (errors, webhooks, reconcile failures) ========== Source: rhods-operator pods in redhat-ods-operator (since-time=2026-07-05T05:14:48Z) --- redhat-ods-operator/rhods-operator-5bb78c9fff-d5wv9 --- {"level":"info","ts":"2026-07-05T06:46:28Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"error occurred during rule evaluation: clusterrole.rbac.authorization.k8s.io \"kueue-batch-user-role\" not found","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c"} --------------------------------------------------------------------------------------------------- ==================== OPERATOR POD LOG: rhods-operator-5bb78c9fff-d5wv9.log ==================== {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:35:25Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"ce76c3e3-9b48-4fee-bdbf-d500a0baa472","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:35:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"9c8f96fd-b062-4b2c-8ee9-d7ecd14ad010","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:10Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"da6d95dc-728e-4d25-a480-b9016e78408b","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"f6db7ef0-9de0-4fb1-b62c-4c7b72996573"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"f6db7ef0-9de0-4fb1-b62c-4c7b72996573"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"f6db7ef0-9de0-4fb1-b62c-4c7b72996573","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"1949ad4e-d874-4bc4-9ad0-e4f2f1f3af8d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"f6db7ef0-9de0-4fb1-b62c-4c7b72996573","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-05T06:36:47Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"2ac9a3e5-3ca5-49df-884f-1415192281e5","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"2ac9a3e5-3ca5-49df-884f-1415192281e5","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"c9c37929-ebec-42d3-a145-72c26877a722","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"f6db7ef0-9de0-4fb1-b62c-4c7b72996573","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.ensureMaasClusterConfigControllerRef"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"f6db7ef0-9de0-4fb1-b62c-4c7b72996573","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"f6db7ef0-9de0-4fb1-b62c-4c7b72996573","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"f6db7ef0-9de0-4fb1-b62c-4c7b72996573","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9e263ba4-73eb-44c8-b061-c55be932ff60"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9e263ba4-73eb-44c8-b061-c55be932ff60"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9e263ba4-73eb-44c8-b061-c55be932ff60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"332b210c-9552-48a3-a5e2-d28607ddd61c"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"332b210c-9552-48a3-a5e2-d28607ddd61c"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"332b210c-9552-48a3-a5e2-d28607ddd61c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"332b210c-9552-48a3-a5e2-d28607ddd61c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"332b210c-9552-48a3-a5e2-d28607ddd61c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"332b210c-9552-48a3-a5e2-d28607ddd61c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"332b210c-9552-48a3-a5e2-d28607ddd61c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9e263ba4-73eb-44c8-b061-c55be932ff60","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9e263ba4-73eb-44c8-b061-c55be932ff60","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.ensureMaasClusterConfigControllerRef"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9e263ba4-73eb-44c8-b061-c55be932ff60","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9e263ba4-73eb-44c8-b061-c55be932ff60","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"9e263ba4-73eb-44c8-b061-c55be932ff60","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"332b210c-9552-48a3-a5e2-d28607ddd61c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:36:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"332b210c-9552-48a3-a5e2-d28607ddd61c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"reconcile","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"apply","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createGatewayInfrastructure"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createKubeAuthProxyInfrastructure"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createEnvoyFilter"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createNetworkPolicy"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createOCPRoutes"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.createDashboardRedirects"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/gateway.syncGatewayConfigStatus"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:37:40Z","msg":"Executing action","controller":"gatewayconfig","controllerGroup":"services.platform.opendatahub.io","controllerKind":"GatewayConfig","GatewayConfig":{"name":"default-gateway"},"namespace":"","name":"default-gateway","reconcileID":"324085dc-9de4-4418-8ac5-7c18122fb479","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:38:44Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-05T06:39:01Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-05T06:39:38Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"c6d56562-055d-419a-977c-affaad3ea8bc"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"c6d56562-055d-419a-977c-affaad3ea8bc"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"c6d56562-055d-419a-977c-affaad3ea8bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"c6d56562-055d-419a-977c-affaad3ea8bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"a366d152-12be-4e5f-a2ff-6ccccda18fb5","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"cc7e7f0f-cd33-4ef1-8f9b-b068c225780f","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"cc7e7f0f-cd33-4ef1-8f9b-b068c225780f","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"7383c705-c1be-4aba-8d1f-adc60917b81d","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"c6d56562-055d-419a-977c-affaad3ea8bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.ensureMaasClusterConfigControllerRef"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"c6d56562-055d-419a-977c-affaad3ea8bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"c6d56562-055d-419a-977c-affaad3ea8bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"c6d56562-055d-419a-977c-affaad3ea8bc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"6c80697b-9c33-4d76-ad03-c671006166c0"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"6c80697b-9c33-4d76-ad03-c671006166c0"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"6c80697b-9c33-4d76-ad03-c671006166c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3340eaa1-2332-452b-b66e-43aa8d4a3c94"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3340eaa1-2332-452b-b66e-43aa8d4a3c94"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3340eaa1-2332-452b-b66e-43aa8d4a3c94","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3340eaa1-2332-452b-b66e-43aa8d4a3c94","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3340eaa1-2332-452b-b66e-43aa8d4a3c94","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3340eaa1-2332-452b-b66e-43aa8d4a3c94","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3340eaa1-2332-452b-b66e-43aa8d4a3c94","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3340eaa1-2332-452b-b66e-43aa8d4a3c94","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:47Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"6c80697b-9c33-4d76-ad03-c671006166c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:48Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c8410f6b-df00-45e7-9c6f-38acc9320049","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"c8410f6b-df00-45e7-9c6f-38acc9320049","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"6c80697b-9c33-4d76-ad03-c671006166c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.ensureMaasClusterConfigControllerRef"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"6c80697b-9c33-4d76-ad03-c671006166c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"6c80697b-9c33-4d76-ad03-c671006166c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"6c80697b-9c33-4d76-ad03-c671006166c0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"3340eaa1-2332-452b-b66e-43aa8d4a3c94","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d1c4de2-07d6-4ae6-9c88-678416e8c364"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d1c4de2-07d6-4ae6-9c88-678416e8c364"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d1c4de2-07d6-4ae6-9c88-678416e8c364","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d1c4de2-07d6-4ae6-9c88-678416e8c364","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d1c4de2-07d6-4ae6-9c88-678416e8c364","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d1c4de2-07d6-4ae6-9c88-678416e8c364","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d1c4de2-07d6-4ae6-9c88-678416e8c364","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d1c4de2-07d6-4ae6-9c88-678416e8c364","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:41:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"4d1c4de2-07d6-4ae6-9c88-678416e8c364","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:44:50Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-05T06:46:24Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-y531q","reconcileID":"bbffe9d4-f668-4dae-a2b5-612c6e748f1e"} {"level":"info","ts":"2026-07-05T06:46:24Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-y531q","reconcileID":"aac48049-9efa-4aa3-aede-5e80fe42f2dc"} {"level":"info","ts":"2026-07-05T06:46:24Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-y531q","reconcileID":"bffec3a7-21a0-4b3a-9499-5c3646fa0c97"} {"level":"info","ts":"2026-07-05T06:46:25Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-1e207","reconcileID":"5676a15b-cf7f-4a2e-9fb4-0cb4a8c21f52"} {"level":"info","ts":"2026-07-05T06:46:25Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-1e207","reconcileID":"db810a21-4b6f-4405-b866-7246577fbf66"} {"level":"info","ts":"2026-07-05T06:46:25Z","msg":"Adding CA bundle configmap","controller":"cert-configmap-generator-controller","namespace":"","name":"test-ns-1e207","reconcileID":"f401ea0d-12c2-4085-804a-24835f55d5e0"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/dependency.(*action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-05T06:46:28Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"365f1d41-cb4f-44e2-98bf-7aad54445c22","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/dependency.(*action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-05T06:46:28Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm","msg":"error occurred during rule evaluation: clusterrole.rbac.authorization.k8s.io \"kueue-batch-user-role\" not found","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"63531fc9-d891-406a-8aae-63d7cd2409ae","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"reconcile","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"apply","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.checkPreConditions"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.initialize"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/dependency.(*action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/releases.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/kustomize.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageDefaultKueueResourcesAction"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding"} {"level":"info","ts":"2026-07-05T06:46:28Z","logger":"action.github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.manageKueueAdminRoleBinding","msg":"Successfully managed kueue admin role binding","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","adminGroups":["rhods-admins"]} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.(*componentHandler).NewComponentReconciler.func1"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/kueue.configureClusterQueueViewerRoleAction"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"kueue","controllerGroup":"components.platform.opendatahub.io","controllerKind":"Kueue","Kueue":{"name":"default-kueue"},"namespace":"","name":"default-kueue","reconcileID":"af5c1aa0-880d-46b6-8589-a7ddb533b240","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:46:28Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"13815c1b-ee43-48d8-876c-bf2a59d8137c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36e35587-0a30-4522-ba3c-d86d2e9fafdc"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36e35587-0a30-4522-ba3c-d86d2e9fafdc"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36e35587-0a30-4522-ba3c-d86d2e9fafdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"f15a2f79-5eea-4a07-ae20-a96b24a373b0","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36e35587-0a30-4522-ba3c-d86d2e9fafdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-05T06:46:48Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"53622242-cdf0-4194-b8c5-f2ecd167795f","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"53622242-cdf0-4194-b8c5-f2ecd167795f","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"84edeb0c-4c74-4dc1-9b85-0ccd48323e61","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36e35587-0a30-4522-ba3c-d86d2e9fafdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.ensureMaasClusterConfigControllerRef"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36e35587-0a30-4522-ba3c-d86d2e9fafdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36e35587-0a30-4522-ba3c-d86d2e9fafdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36e35587-0a30-4522-ba3c-d86d2e9fafdc","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36a0ec12-38f2-4e50-aff9-ce209593408c"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36a0ec12-38f2-4e50-aff9-ce209593408c"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36a0ec12-38f2-4e50-aff9-ce209593408c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6ee342d5-553d-45f2-888b-465186300b9a"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6ee342d5-553d-45f2-888b-465186300b9a"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6ee342d5-553d-45f2-888b-465186300b9a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6ee342d5-553d-45f2-888b-465186300b9a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6ee342d5-553d-45f2-888b-465186300b9a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6ee342d5-553d-45f2-888b-465186300b9a","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6ee342d5-553d-45f2-888b-465186300b9a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36a0ec12-38f2-4e50-aff9-ce209593408c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36a0ec12-38f2-4e50-aff9-ce209593408c","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.ensureMaasClusterConfigControllerRef"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36a0ec12-38f2-4e50-aff9-ce209593408c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36a0ec12-38f2-4e50-aff9-ce209593408c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"36a0ec12-38f2-4e50-aff9-ce209593408c","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6ee342d5-553d-45f2-888b-465186300b9a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:46:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6ee342d5-553d-45f2-888b-465186300b9a","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:47:05Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-05T06:47:38Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"377f6689-4095-45ea-a215-9ad7cf6ae8ed"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"377f6689-4095-45ea-a215-9ad7cf6ae8ed"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"377f6689-4095-45ea-a215-9ad7cf6ae8ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"377f6689-4095-45ea-a215-9ad7cf6ae8ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"dbb5259a-aee3-4a3f-a156-297c27c4b775","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","logger":"DSCInitialization","msg":"Reconciling DSCInitialization.","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"34ee9c9c-f865-44df-9737-f722af2fea7e","DSCInitialization Request.Name":"default-dsci"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Application namespace set in DSCI not found, creating it with labels","controller":"dscinitialization","controllerGroup":"dscinitialization.opendatahub.io","controllerKind":"DSCInitialization","DSCInitialization":{"name":"default-dsci"},"namespace":"","name":"default-dsci","reconcileID":"34ee9c9c-f865-44df-9737-f722af2fea7e","name":"redhat-ods-applications"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"reconcile","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"apply","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.addMonitoringCapability"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringAdmissionPolicies"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployMonitoringStackWithQuerierAndRestrictions"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployTracingStack"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployAlerting"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployOpenTelemetryCollector"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPerses"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesTempoIntegration"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployPersesPrometheusIntegration"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.deployNodeMetricsEndpoint"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/render/template.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/services/monitoring.syncPrometheusWebTLSCA"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"monitoring","controllerGroup":"services.platform.opendatahub.io","controllerKind":"Monitoring","Monitoring":{"name":"default-monitoring"},"namespace":"","name":"default-monitoring","reconcileID":"31ae35f8-7fd7-48d2-b30e-12b38c8b96aa","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"377f6689-4095-45ea-a215-9ad7cf6ae8ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.ensureMaasClusterConfigControllerRef"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"377f6689-4095-45ea-a215-9ad7cf6ae8ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"377f6689-4095-45ea-a215-9ad7cf6ae8ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"377f6689-4095-45ea-a215-9ad7cf6ae8ed","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"reconcile","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"d155144b-54a7-4b4b-b1c9-0f1b6e7bacfe"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"apply","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"d155144b-54a7-4b4b-b1c9-0f1b6e7bacfe"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"d155144b-54a7-4b4b-b1c9-0f1b6e7bacfe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.renderMaasOperatorInstall"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"reconcile","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c47df51-b8d9-48de-9f27-14d7e3c29030"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"apply","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c47df51-b8d9-48de-9f27-14d7e3c29030"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c47df51-b8d9-48de-9f27-14d7e3c29030","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.initialize"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c47df51-b8d9-48de-9f27-14d7e3c29030","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.checkPreConditions"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c47df51-b8d9-48de-9f27-14d7e3c29030","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.updateStatus"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c47df51-b8d9-48de-9f27-14d7e3c29030","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/datasciencecluster.provisionComponents"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c47df51-b8d9-48de-9f27-14d7e3c29030","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c47df51-b8d9-48de-9f27-14d7e3c29030","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"d155144b-54a7-4b4b-b1c9-0f1b6e7bacfe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/deploy.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"d155144b-54a7-4b4b-b1c9-0f1b6e7bacfe","action":"github.com/opendatahub-io/opendatahub-operator/v2/internal/controller/components/modelsasservice.ensureMaasClusterConfigControllerRef"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"d155144b-54a7-4b4b-b1c9-0f1b6e7bacfe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/status/deployments.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"d155144b-54a7-4b4b-b1c9-0f1b6e7bacfe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/actions/gc.(*Action).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"modelsasservice","controllerGroup":"components.platform.opendatahub.io","controllerKind":"ModelsAsService","ModelsAsService":{"name":"default-modelsasservice"},"namespace":"","name":"default-modelsasservice","reconcileID":"d155144b-54a7-4b4b-b1c9-0f1b6e7bacfe","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:51:48Z","msg":"Executing action","controller":"datasciencecluster","controllerGroup":"datasciencecluster.opendatahub.io","controllerKind":"DataScienceCluster","DataScienceCluster":{"name":"default-dsc"},"namespace":"","name":"default-dsc","reconcileID":"6c47df51-b8d9-48de-9f27-14d7e3c29030","action":"github.com/opendatahub-io/opendatahub-operator/v2/pkg/controller/reconciler.(*dynamicWatchAction).run-fm"} {"level":"info","ts":"2026-07-05T06:52:08Z","logger":"controller-runtime.cache","msg":"Warning: This version is deprecated. Use v1beta2 instead."} ==================== OPERATOR POD LOG: rhods-operator-5bb78c9fff-l66pr.log ==================== ==================== OPERATOR POD LOG: rhods-operator-5bb78c9fff-vf6z7.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/dashboard-redirect-74cb9949d4-j7pj9.log ==================== ==================== WORKLOAD POD LOG: redhat-ods-applications/data-science-pipelines-operator-controller-manager-99b58762tnpn.log ==================== 2026-07-05T05:51:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:12Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:12Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:12Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:12Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:12Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:12Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:12Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:12Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:12Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:13Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:13Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:13Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:13Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:13Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:13Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:13Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:13Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:13Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:13Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:13Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:13Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:13Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:13Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:13Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:13Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:14Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:14Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:14Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:14Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:14Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:14Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:14Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:14Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:14Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:14Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:14Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:14Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:14Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:14Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:14Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:14Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:15Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:15Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:15Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:15Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:15Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:15Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:15Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:15Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:15Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:15Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:15Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:15Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:15Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:15Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:15Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:15Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:16Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:16Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:16Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:16Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:16Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:16Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:16Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:16Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:16Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:16Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:16Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:16Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:16Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:16Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:16Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:16Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:16Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:21Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:21Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:21Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:21Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:21Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:21Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:22Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:22Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:22Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:22Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:22Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:22Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:22Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:22Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:22Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:22Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:22Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:22Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:22Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:22Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:22Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:22Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:25Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:26Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:26Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:26Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:26Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:26Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:26Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:26Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:26Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:27Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Could not connect to (minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000), Error: Get "http://minio-dspa.dspa-test-p2fegabi.svc.cluster.local:9000/mlpipeline/?location=": dial tcp 172.30.156.93:9000: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Object Storage Health Check Failed {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:27Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:27Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:27Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:27Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:27Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:27Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:27Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:27Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:33Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:33Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:33Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:33Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:33Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:33Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:33Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Unable to connect to Database: dial tcp 172.30.202.131:3306: connect: connection refused {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Health check for Database or Object Store failed, retrying in 20 seconds. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:33Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:33Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:33Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:33Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:33Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:33Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:33Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:33Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:33Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:34Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:34Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:34Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:34Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:35Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:35Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:35Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:35Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:35Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:35Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:35Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:35Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"dspa-test-p2fegabi"}, "namespace": "dspa-test-p2fegabi", "name": "dspa", "reconcileID": "5052db7a-1e76-4ed8-a2a4-8d71580ef6e8", "error": "secret containing the certificate for MLMD gRPC Server was not created yet"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:51:35Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:35Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:36Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:36Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:36Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:36Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:36Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:37Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:37Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:37Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:37Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:37Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:37Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:37Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:37Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:37Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:38Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:38Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:38Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:38Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:38Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:38Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:38Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:38Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:38Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:39Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:39Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:39Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:39Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:39Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:39Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:39Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:39Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:39Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:40Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:40Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:40Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:40Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:40Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:40Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:40Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:40Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:40Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:41Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:41Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:41Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:41Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:41Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:41Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:41Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:41Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:41Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:42Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:42Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:42Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:42Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:42Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:42Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:42Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:42Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:42Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:42Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:51:42Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:42Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:42Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:51:43Z ERROR Reconciler error {"controller": "datasciencepipelinesapplication", "controllerGroup": "datasciencepipelinesapplications.opendatahub.io", "controllerKind": "DataSciencePipelinesApplication", "DataSciencePipelinesApplication": {"name":"dspa","namespace":"dspa-test-p2fegabi"}, "namespace": "dspa-test-p2fegabi", "name": "dspa", "reconcileID": "cc12c832-f1a7-44e8-933d-1048758be648", "error": "Operation cannot be fulfilled on deployments.apps \"ds-pipeline-persistenceagent-dspa\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:495 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:51:43Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:43Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:43Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:43Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:43Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:43Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:43Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:43Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:43Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:43Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:44Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:44Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:45Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:45Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:45Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:45Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:45Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:45Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:45Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:45Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:46Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:46Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO MLMD not specified, but is a required component for Pipelines. Including MLMD with default specs. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Secret [ds-pipeline-db-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Secret [ds-pipeline-s3-dspa] already exists, using stored value. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying mariaDB resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying Database Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO No S3 storage credential reference provided, so using managed secret {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying object storage resources. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying storage Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Performing Database Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Database Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Performing Object Storage Health Check {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Object Storage Health Check Successful {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying Common Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Generating Sample Config {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying APIServer Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying PersistenceAgent Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying ScheduledWorkflow Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying WorkflowController Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Applying ML-Metadata (MLMD) Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Finished applying MLMD Resources {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:51:47Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:51:47Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "True"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:17Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:17Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:17Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:17Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:18Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:18Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:18Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:18Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:18Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:18Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:18Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:18Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:18Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:18Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:18Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:18Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:18Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:18Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:18Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:18Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:27Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:27Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:27Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:27Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:27Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:27Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:27Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:27Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:27Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:27Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:27Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:47Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:47Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:47Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:47Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:47Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:47Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:47Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:47Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:47Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:47Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:47Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:47Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:47Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:47Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:47Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:47Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:47Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:48Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:48Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:48Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:48Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:48Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:48Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:48Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:48Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:48Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:54Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:54Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:54Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:54Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:54Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:54Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:54Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:54Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:54Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:54Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Found global CA Bundle odh-trusted-ca-bundle present in this namespace dspa-test-p2fegabi, this bundle will be included in external tls connections. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"dsp-trusted-ca-dspa\" is forbidden: unable to create new content in namespace dspa-test-p2fegabi because it is being terminated"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:54Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:54Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:54Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:54Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:54Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:54Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:54Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:54Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:55Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:55Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:55Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:55Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:55Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:55Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:55Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:55Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:55Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:55Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:55Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:55Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:55Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:55Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:55Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:55Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:55Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:55Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:55Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:55Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:55Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:55Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:56Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:56Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:56Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:56Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:56Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:56Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:56Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:56Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:56Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:56Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:56Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:58:56Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:58:56Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:58:56Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:58:56Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:01Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:01Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:01Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:01Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:01Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:01Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:01Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:01Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:01Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:01Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:01Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:01Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:01Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:01Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:02Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:02Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:02Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:02Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:02Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:02Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:02Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:02Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:02Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:02Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:02Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:02Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:02Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:02Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:02Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:02Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Encountered error when attempting to fetch ConfigMap: [openshift-service-ca.crt]. Error: &ConfigMap{ObjectMeta:{ 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},Data:map[string]string{},BinaryData:map[string][]byte{},Immutable:nil,} {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z ERROR Encountered error when parsing CR {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", "error": "configmaps \"openshift-service-ca.crt\" not found"} github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile /workspace/controllers/dspipeline_controller.go:311 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313 2026-07-05T05:59:02Z INFO Publishing Ready Metrics {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} 2026-07-05T05:59:02Z INFO DatabaseAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ObjectStoreAvailable {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO PersistenceAgentReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO WorkflowControllerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO APIServerReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO ScheduledWorkflowReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO MLMDProxyReady {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "Unknown"} 2026-07-05T05:59:02Z INFO Ready {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa", " Status:": "False"} 2026-07-05T05:59:03Z INFO No other DSPAs with PipelineStore 'kubernetes' found. Cleaning up webhook resources. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-05T05:59:03Z INFO Webhook resources cleanup complete. {"namespace": "redhat-ods-applications", "dspa_name": "dspa"} 2026-07-05T05:59:03Z INFO DSPA resources cleaned up. {"namespace": "dspa-test-p2fegabi", "dspa_name": "dspa"} ==================== WORKLOAD POD LOG: redhat-ods-applications/feast-operator-controller-manager-86c6459f75-gp7rl.log ==================== 2026-07-05T05:33:53Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "f2642875-068a-42cf-ac2a-b17c9ad5e9c0", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:33:53Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "1eeba962-4a34-4778-be17-943a5f67b978", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:33:53Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "c22736c9-2b45-47cb-96ad-5c459c201a7d", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:33:54Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "ccf895cf-ff87-440f-83c4-133893c3813a", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:33:57Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "4490c871-d842-44c9-a5cf-fbefd59e7599", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:33:57Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "12703060-12c0-4fac-a04c-e967fa1b6df9", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:01Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "f3139819-740e-44a5-b9b8-6547cbd32743", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:03Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "eac3b60c-8f90-4677-ac20-ab8bf8adef78", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:10Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "71c0d25c-7e49-4aa0-8aec-9f5d20c10562", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:15Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "5412ddf9-4475-46de-ad21-2a8f18fe7cb7", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:15Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "7a1d530e-48c0-4771-b78e-c7ebc9f8b734", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:16Z DEBUG Notebook is being deleted, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "ff20ad34-0743-4f95-bcde-e203f5ca0b6b"} 2026-07-05T05:34:16Z DEBUG Notebook not found, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-odh-notebook","namespace":"test-odh-notebook"}, "namespace": "test-odh-notebook", "name": "test-odh-notebook", "reconcileID": "d148be41-a66d-48ba-a1b5-cab81f3cb56b"} 2026-07-05T05:34:28Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "1c1a8ee4-0c9f-4e10-8323-c30944cfa97b", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:28Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "ff933dd1-3438-49d1-84af-27e3a1fc3c2e", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:28Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "5e63c94d-b1f9-4b95-8c3e-40f9c90a5135", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:29Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "25f58caf-5cfd-4eec-b088-5af1b31f6143", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:32Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "2c186b7f-02a7-4af3-8f99-324b37d9d002", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:32Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "3df69d81-11c5-45fc-a3c2-83231b88ed4a", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:42Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "46002935-4b4f-49c6-a9ad-a7a3d357aa20", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:43Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "f84ccde1-cace-4e8f-9853-e46aafa82331", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:50Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "27aa8f80-317e-4ee2-a01f-8690aa7b5cf2", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:55Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "cfb197f6-b325-43f3-b598-4a9f3e920003", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:55Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "734e8ce4-9491-4f2d-8c8d-ab2dd68c1669", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T05:34:56Z DEBUG Notebook is being deleted, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "95b2b984-fdc1-42dc-b84c-dbb09c0f6a22"} 2026-07-05T05:34:56Z DEBUG Notebook not found, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"test-auth-notebook","namespace":"test-auth-notebook"}, "namespace": "test-auth-notebook", "name": "test-auth-notebook", "reconcileID": "c030c783-6118-4184-a06e-fc71d0e6c0bf"} 2026-07-05T06:21:12Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "ff76ece1-a453-488a-aa8f-1190010031d2", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:12Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "092a4fa0-cfa4-4b7d-876d-5160136c79f0", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:13Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "f6f827f5-eeb1-4c42-b0be-0a0215d4c902", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:13Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "6b4a455a-193d-40e2-928f-60fe60b9b30c", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:13Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "ac40296f-c973-4f66-905d-65c42ecdcf00", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:13Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "ad0e74ea-7867-48b6-b8a7-dbc2f850a166", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:22Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "f8696848-6396-454d-87be-62f9c7878143", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:23Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "006b2476-f68c-4bbf-a8a9-06bd79ee76c6", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:29Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "93f47915-5463-4fc0-bf51-1ae7921b716f", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:34Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "a5b61bf3-4ed7-479a-a1c6-07b544b39fd5", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:34Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "f3ff5fb8-dec3-44ab-924a-c4f1ce2f3883", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:48Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "d4fa6ba7-3bd7-434e-9e14-a40d0d15b292", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:48Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "5cc33966-9e99-41f1-9af3-fc31443ec533", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:49Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "0519b7e7-7103-4146-ab83-d1eec7dddee4", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:21:49Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "e29bfd76-4aec-49bd-a877-d007719aa861", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:22:22Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:22:22Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "ConfigMap": "feast-test-s3-client-ca", "operation": "created"} 2026-07-05T06:22:22Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "Service": "feast-test-s3-online", "operation": "created"} 2026-07-05T06:22:22Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "Service": "feast-test-s3-registry", "operation": "created"} 2026-07-05T06:22:22Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "Service": "feast-test-s3-registry-rest", "operation": "created"} 2026-07-05T06:22:22Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "ServiceAccount": "feast-test-s3", "operation": "created"} 2026-07-05T06:22:22Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "ConfigMap": "feast-test-s3-intra-comm", "operation": "created"} 2026-07-05T06:22:22Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7"} 2026-07-05T06:22:22Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7"} 2026-07-05T06:22:22Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7"} 2026-07-05T06:22:22Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "Deployment": "feast-test-s3", "operation": "created"} 2026-07-05T06:22:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7"} 2026-07-05T06:22:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "ConfigMap": "feast-test-s3-client", "operation": "created"} 2026-07-05T06:22:23Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:23Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:23Z INFO Successfully reconciled namespace registry ConfigMap {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "ConfigMap": "feast-configs-registry", "Namespace": "redhat-ods-applications", "operation": "updated"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "Role": "feast-test-s3-cronjob", "operation": "created"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "RoleBinding": "feast-test-s3-cronjob", "operation": "created"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7", "CronJob": "feast-test-s3", "operation": "created"} 2026-07-05T06:22:23Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01b02ad5-03c9-4a76-8cbe-bf733c10c8e7"} 2026-07-05T06:22:23Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:22:23Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:22:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b"} 2026-07-05T06:22:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b"} 2026-07-05T06:22:23Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b"} 2026-07-05T06:22:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b"} 2026-07-05T06:22:23Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:23Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:23Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b"} 2026-07-05T06:22:23Z INFO FeatureStore object modified, retry syncing status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b"} 2026-07-05T06:22:23Z ERROR Error updating the FeatureStore status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b", "error": "Operation cannot be fulfilled on featurestores.feast.dev \"test-s3\": the object has been modified; please apply your changes to the latest version and try again"} github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:114 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:22:23Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b"} 2026-07-05T06:22:23Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "62847877-1197-4a3d-a39d-a264d639d73b", "error": "Operation cannot be fulfilled on featurestores.feast.dev \"test-s3\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:22:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4"} 2026-07-05T06:22:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4"} 2026-07-05T06:22:23Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4"} 2026-07-05T06:22:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4"} 2026-07-05T06:22:23Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:23Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:23Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "73c1a885-0ec0-4318-ae76-d60e52e761c4"} 2026-07-05T06:22:24Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:22:24Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:22:24Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:22:24Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056"} 2026-07-05T06:22:24Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056"} 2026-07-05T06:22:24Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056"} 2026-07-05T06:22:24Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:24Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056"} 2026-07-05T06:22:24Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056"} 2026-07-05T06:22:24Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:24Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:24Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:24Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "964d2cb5-b6b6-4b99-b29b-7aaf5d14d056"} 2026-07-05T06:22:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:22:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:22:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:22:29Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912"} 2026-07-05T06:22:29Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912"} 2026-07-05T06:22:29Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912"} 2026-07-05T06:22:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:29Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912"} 2026-07-05T06:22:29Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912"} 2026-07-05T06:22:29Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:29Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:29Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f19abf18-ff46-4e20-98f1-def3b887e912"} 2026-07-05T06:22:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:22:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:22:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:22:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65"} 2026-07-05T06:22:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65"} 2026-07-05T06:22:34Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65"} 2026-07-05T06:22:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65"} 2026-07-05T06:22:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65"} 2026-07-05T06:22:34Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:34Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:34Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4320194b-d3d8-4dbf-9135-1e23eb4a4d65"} 2026-07-05T06:22:34Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:22:34Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:22:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:22:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:22:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:22:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963"} 2026-07-05T06:22:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963"} 2026-07-05T06:22:34Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963"} 2026-07-05T06:22:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963"} 2026-07-05T06:22:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963"} 2026-07-05T06:22:34Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:34Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:35Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af29131d-92e6-42db-b558-fc4069c0b963"} 2026-07-05T06:22:39Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:22:39Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:22:39Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:22:39Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f"} 2026-07-05T06:22:39Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f"} 2026-07-05T06:22:39Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f"} 2026-07-05T06:22:39Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:39Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f"} 2026-07-05T06:22:39Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f"} 2026-07-05T06:22:39Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:39Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:39Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:39Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "723ac0d6-e4bb-4318-8d80-73e21eedd77f"} 2026-07-05T06:22:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:22:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:22:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:22:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa"} 2026-07-05T06:22:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa"} 2026-07-05T06:22:41Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa"} 2026-07-05T06:22:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa"} 2026-07-05T06:22:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa"} 2026-07-05T06:22:41Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:41Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:41Z INFO FeatureStore installation complete {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa"} 2026-07-05T06:22:41Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:22:41Z INFO Added Feast discovery label to namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:22:41Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:41Z INFO Successfully added feature store to namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "namespace": "test-fs-ui-proj-108259", "clientConfig": "feast-test-s3-client", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:42Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:22:42Z DEBUG Registry permissions response {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "body": "{\"permissions\":[],\"pagination\":{}}"} 2026-07-05T06:22:42Z DEBUG Extracted permission policies {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "permissionsCount": 0, "policiesCount": 0} 2026-07-05T06:22:42Z DEBUG Auto-access prerequisites missing or registry unreachable; cleaning up stale auto-access RBAC {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ca1b1c-63ce-443b-a0eb-18a0e955abaa", "policies": 0, "clientConfigMapSet": true, "fetchError": false} 2026-07-05T06:22:42Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:22:42Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:22:42Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:22:42Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28"} 2026-07-05T06:22:42Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28"} 2026-07-05T06:22:42Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28"} 2026-07-05T06:22:42Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:42Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28"} 2026-07-05T06:22:42Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28"} 2026-07-05T06:22:42Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:42Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:42Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:42Z INFO FeatureStore installation complete {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28"} 2026-07-05T06:22:42Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:42Z INFO Successfully added feature store to namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "namespace": "test-fs-ui-proj-108259", "clientConfig": "feast-test-s3-client", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:42Z DEBUG Registry permissions response {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "body": "{\"permissions\":[],\"pagination\":{}}"} 2026-07-05T06:22:42Z DEBUG Extracted permission policies {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "permissionsCount": 0, "policiesCount": 0} 2026-07-05T06:22:42Z DEBUG Auto-access prerequisites missing or registry unreachable; cleaning up stale auto-access RBAC {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "92ed01d8-e435-40ab-98dc-2f7db7cfca28", "policies": 0, "clientConfigMapSet": true, "fetchError": false} 2026-07-05T06:22:45Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:22:45Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:22:45Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:22:45Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65"} 2026-07-05T06:22:45Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65"} 2026-07-05T06:22:45Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65"} 2026-07-05T06:22:45Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:45Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65"} 2026-07-05T06:22:45Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65"} 2026-07-05T06:22:45Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:45Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:45Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:22:45Z INFO FeatureStore installation complete {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65"} 2026-07-05T06:22:45Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "namespace": "redhat-ods-applications"} 2026-07-05T06:22:45Z INFO Successfully added feature store to namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "namespace": "test-fs-ui-proj-108259", "clientConfig": "feast-test-s3-client", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:22:45Z DEBUG Registry permissions response {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "body": "{\"permissions\":[],\"pagination\":{}}"} 2026-07-05T06:22:45Z DEBUG Extracted permission policies {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "permissionsCount": 0, "policiesCount": 0} 2026-07-05T06:22:45Z DEBUG Auto-access prerequisites missing or registry unreachable; cleaning up stale auto-access RBAC {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "811b4184-72c1-491c-9d64-355956300f65", "policies": 0, "clientConfigMapSet": true, "fetchError": false} 2026-07-05T06:23:05Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:05Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:05Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:05Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa"} 2026-07-05T06:23:05Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa"} 2026-07-05T06:23:05Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa"} 2026-07-05T06:23:05Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:05Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa"} 2026-07-05T06:23:05Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa"} 2026-07-05T06:23:05Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:05Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:05Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:05Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a544a9a-b8b2-4bd1-9624-853de60ca4fa"} 2026-07-05T06:23:05Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:23:05Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:23:05Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:05Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:05Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:05Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd"} 2026-07-05T06:23:05Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd"} 2026-07-05T06:23:05Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd"} 2026-07-05T06:23:05Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:05Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd"} 2026-07-05T06:23:06Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd"} 2026-07-05T06:23:06Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:06Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:06Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:06Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82fe5c91-36c3-49de-b5f6-a16d2ada8fbd"} 2026-07-05T06:23:10Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:10Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:10Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:10Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8"} 2026-07-05T06:23:10Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8"} 2026-07-05T06:23:10Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8"} 2026-07-05T06:23:10Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:10Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8"} 2026-07-05T06:23:10Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8"} 2026-07-05T06:23:10Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:10Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:10Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:10Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1cc0e282-35c6-4699-b528-d140ec1633a8"} 2026-07-05T06:23:16Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:16Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:16Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:16Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d"} 2026-07-05T06:23:16Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d"} 2026-07-05T06:23:16Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d"} 2026-07-05T06:23:16Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:16Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d"} 2026-07-05T06:23:16Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d"} 2026-07-05T06:23:16Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:16Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:16Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:16Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bd555972-4ebc-474e-a56a-30432c10777d"} 2026-07-05T06:23:21Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:21Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:21Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:21Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c"} 2026-07-05T06:23:21Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c"} 2026-07-05T06:23:21Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c"} 2026-07-05T06:23:21Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:21Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c"} 2026-07-05T06:23:21Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c"} 2026-07-05T06:23:21Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:21Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:21Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:21Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f276fa94-e69f-4cf7-bbc6-030ff6faad4c"} 2026-07-05T06:23:26Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:26Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:26Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:26Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a"} 2026-07-05T06:23:26Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a"} 2026-07-05T06:23:26Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a"} 2026-07-05T06:23:26Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:26Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a"} 2026-07-05T06:23:26Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a"} 2026-07-05T06:23:26Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:26Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:26Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:26Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2f2a7f2f-62ee-4cd2-8411-4b7323069d6a"} 2026-07-05T06:23:31Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:31Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:31Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:31Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447"} 2026-07-05T06:23:31Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447"} 2026-07-05T06:23:31Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447"} 2026-07-05T06:23:31Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:32Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447"} 2026-07-05T06:23:32Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447"} 2026-07-05T06:23:32Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:32Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:32Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:32Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8e1b5f89-9457-4fb8-8a60-9b3166ed5447"} 2026-07-05T06:23:37Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:37Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:37Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:37Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0"} 2026-07-05T06:23:37Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0"} 2026-07-05T06:23:37Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0"} 2026-07-05T06:23:37Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:37Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0"} 2026-07-05T06:23:37Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0"} 2026-07-05T06:23:37Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:37Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:37Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:37Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32ba095d-61bd-4c4d-bea5-95d53a58c0e0"} 2026-07-05T06:23:42Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:42Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:42Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:42Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6"} 2026-07-05T06:23:42Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6"} 2026-07-05T06:23:42Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6"} 2026-07-05T06:23:42Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:42Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6"} 2026-07-05T06:23:42Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6"} 2026-07-05T06:23:42Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:42Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:42Z ERROR Error deploying the FeatureStore client service {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6", "error": "rolebindings.rbac.authorization.k8s.io \"feast-test-s3-cronjob\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).setFeastServiceCondition /opt/app-root/src/internal/controller/services/services.go:1205 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).deployCronJob /opt/app-root/src/internal/controller/services/cronjob.go:23 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:101 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:42Z INFO Error: rolebindings.rbac.authorization.k8s.io "feast-test-s3-cronjob" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6"} 2026-07-05T06:23:42Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6"} 2026-07-05T06:23:42Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2ef39bb5-498d-4660-bda1-799c03e9e6f6", "error": "rolebindings.rbac.authorization.k8s.io \"feast-test-s3-cronjob\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:42Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:23:42Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:23:43Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:43Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:43Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:43Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97"} 2026-07-05T06:23:43Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97"} 2026-07-05T06:23:43Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97"} 2026-07-05T06:23:43Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:43Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97"} 2026-07-05T06:23:43Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97"} 2026-07-05T06:23:43Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:43Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:43Z ERROR Error deploying the FeatureStore client service {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97", "error": "rolebindings.rbac.authorization.k8s.io \"feast-test-s3-cronjob\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).setFeastServiceCondition /opt/app-root/src/internal/controller/services/services.go:1205 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).deployCronJob /opt/app-root/src/internal/controller/services/cronjob.go:23 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:101 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:43Z INFO Error: rolebindings.rbac.authorization.k8s.io "feast-test-s3-cronjob" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97"} 2026-07-05T06:23:43Z INFO FeatureStore object modified, retry syncing status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97"} 2026-07-05T06:23:43Z ERROR Error updating the FeatureStore status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97", "error": "Operation cannot be fulfilled on featurestores.feast.dev \"test-s3\": the object has been modified; please apply your changes to the latest version and try again"} github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:114 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:43Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97"} 2026-07-05T06:23:43Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "804ab787-27bb-40ce-b7de-92d61901eb97", "error": "rolebindings.rbac.authorization.k8s.io \"feast-test-s3-cronjob\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:43Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6c8e501d-5706-4c27-9c49-a0e956de1a9a"} 2026-07-05T06:23:43Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6c8e501d-5706-4c27-9c49-a0e956de1a9a"} 2026-07-05T06:23:43Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6c8e501d-5706-4c27-9c49-a0e956de1a9a", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:43Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:23:43Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:23:43Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6b3b8e41-ed4e-4a9b-9a07-80fd06796f5c"} 2026-07-05T06:23:43Z INFO FeatureStore object modified, retry syncing status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6b3b8e41-ed4e-4a9b-9a07-80fd06796f5c"} 2026-07-05T06:23:43Z ERROR Error updating the FeatureStore status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6b3b8e41-ed4e-4a9b-9a07-80fd06796f5c", "error": "Operation cannot be fulfilled on featurestores.feast.dev \"test-s3\": the object has been modified; please apply your changes to the latest version and try again"} github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:114 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:43Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6b3b8e41-ed4e-4a9b-9a07-80fd06796f5c"} 2026-07-05T06:23:43Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6b3b8e41-ed4e-4a9b-9a07-80fd06796f5c", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:43Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f1235a02-2b5d-42e0-a053-b53a559c4dca"} 2026-07-05T06:23:43Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f1235a02-2b5d-42e0-a053-b53a559c4dca"} 2026-07-05T06:23:43Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f1235a02-2b5d-42e0-a053-b53a559c4dca", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:43Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b72cd3ea-dedd-4095-98a4-87344b3af6b0"} 2026-07-05T06:23:43Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b72cd3ea-dedd-4095-98a4-87344b3af6b0"} 2026-07-05T06:23:43Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b72cd3ea-dedd-4095-98a4-87344b3af6b0", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:43Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6dda436b-9f14-4fda-8cc6-be0ce55ea897"} 2026-07-05T06:23:43Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6dda436b-9f14-4fda-8cc6-be0ce55ea897"} 2026-07-05T06:23:43Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6dda436b-9f14-4fda-8cc6-be0ce55ea897", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:44Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f542e7b8-0156-44ed-b370-aebd65ae7c6a"} 2026-07-05T06:23:44Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f542e7b8-0156-44ed-b370-aebd65ae7c6a"} 2026-07-05T06:23:44Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f542e7b8-0156-44ed-b370-aebd65ae7c6a", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:44Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c55e1a29-0fe5-40f6-a7c0-585f574d263e"} 2026-07-05T06:23:44Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c55e1a29-0fe5-40f6-a7c0-585f574d263e"} 2026-07-05T06:23:44Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c55e1a29-0fe5-40f6-a7c0-585f574d263e", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:44Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "53606e4f-8fa0-4436-9d46-8b6416b41a2e"} 2026-07-05T06:23:44Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "53606e4f-8fa0-4436-9d46-8b6416b41a2e"} 2026-07-05T06:23:44Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "53606e4f-8fa0-4436-9d46-8b6416b41a2e", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:44Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "960026b5-74e5-47bd-a7d8-8a9dd59cda78"} 2026-07-05T06:23:44Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "960026b5-74e5-47bd-a7d8-8a9dd59cda78"} 2026-07-05T06:23:44Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "960026b5-74e5-47bd-a7d8-8a9dd59cda78", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:44Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:23:44Z DEBUG FeatureStore CR not found, has been deleted {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e82bea97-7ecd-42b5-acbd-7a56ad56910f"} 2026-07-05T06:23:44Z INFO Removed Feast discovery label from namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e82bea97-7ecd-42b5-acbd-7a56ad56910f", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:23:44Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e82bea97-7ecd-42b5-acbd-7a56ad56910f", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:44Z INFO Removing feature store from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e82bea97-7ecd-42b5-acbd-7a56ad56910f", "featureStoreName": "test-s3", "featureStoreNamespace": "test-fs-ui-proj-108259", "clientConfigName": "", "expectedClientConfigName": "feast-test-s3-client"} 2026-07-05T06:23:44Z INFO Removing config from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e82bea97-7ecd-42b5-acbd-7a56ad56910f", "config": "feast-test-s3-client"} 2026-07-05T06:23:44Z INFO Removed empty namespace entry from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e82bea97-7ecd-42b5-acbd-7a56ad56910f", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:23:44Z INFO Updated namespace registry ConfigMap {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e82bea97-7ecd-42b5-acbd-7a56ad56910f", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "remainingConfigs": [], "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:44Z INFO Successfully removed feature store from namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e82bea97-7ecd-42b5-acbd-7a56ad56910f", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:44Z DEBUG FeatureStore CR not found, has been deleted {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "21a869c7-6209-498c-9a07-1374df57e30b"} 2026-07-05T06:23:44Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "21a869c7-6209-498c-9a07-1374df57e30b", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:44Z INFO Removing feature store from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "21a869c7-6209-498c-9a07-1374df57e30b", "featureStoreName": "test-s3", "featureStoreNamespace": "test-fs-ui-proj-108259", "clientConfigName": "", "expectedClientConfigName": "feast-test-s3-client"} 2026-07-05T06:23:44Z DEBUG Namespace not found in registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "21a869c7-6209-498c-9a07-1374df57e30b", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:23:44Z INFO Updated namespace registry ConfigMap {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "21a869c7-6209-498c-9a07-1374df57e30b", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "remainingConfigs": [], "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:44Z INFO Successfully removed feature store from namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "21a869c7-6209-498c-9a07-1374df57e30b", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:52Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:23:52Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "ConfigMap": "feast-test-s3-client-ca", "operation": "created"} 2026-07-05T06:23:52Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "Service": "feast-test-s3-online", "operation": "created"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "Service": "feast-test-s3-registry", "operation": "created"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "Service": "feast-test-s3-registry-rest", "operation": "created"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "ServiceAccount": "feast-test-s3", "operation": "created"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "ConfigMap": "feast-test-s3-intra-comm", "operation": "created"} 2026-07-05T06:23:53Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507"} 2026-07-05T06:23:53Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507"} 2026-07-05T06:23:53Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "Deployment": "feast-test-s3", "operation": "created"} 2026-07-05T06:23:53Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507"} 2026-07-05T06:23:53Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "ConfigMap": "feast-test-s3-client", "operation": "created"} 2026-07-05T06:23:53Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:53Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:53Z INFO Successfully reconciled namespace registry ConfigMap {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "ConfigMap": "feast-configs-registry", "Namespace": "redhat-ods-applications", "operation": "updated"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "Role": "feast-test-s3-cronjob", "operation": "created"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "RoleBinding": "feast-test-s3-cronjob", "operation": "created"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507", "CronJob": "feast-test-s3", "operation": "created"} 2026-07-05T06:23:53Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1b53434e-b7ad-4308-b502-8ad6c1d23507"} 2026-07-05T06:23:53Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:23:53Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:53Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2"} 2026-07-05T06:23:53Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2"} 2026-07-05T06:23:53Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:53Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2"} 2026-07-05T06:23:53Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2"} 2026-07-05T06:23:53Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:53Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:53Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2"} 2026-07-05T06:23:53Z INFO FeatureStore object modified, retry syncing status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2"} 2026-07-05T06:23:53Z ERROR Error updating the FeatureStore status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2", "error": "Operation cannot be fulfilled on featurestores.feast.dev \"test-s3\": the object has been modified; please apply your changes to the latest version and try again"} github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:114 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:53Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2"} 2026-07-05T06:23:53Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "64965d76-9a24-4b4a-a930-48537f0668a2", "error": "Operation cannot be fulfilled on featurestores.feast.dev \"test-s3\": the object has been modified; please apply your changes to the latest version and try again"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:23:53Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:54Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:54Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:54Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629"} 2026-07-05T06:23:54Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629"} 2026-07-05T06:23:54Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629"} 2026-07-05T06:23:54Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:54Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629"} 2026-07-05T06:23:54Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629"} 2026-07-05T06:23:54Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:54Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:54Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:54Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8bb17b64-30f2-46fe-9710-c20ece447629"} 2026-07-05T06:23:54Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:54Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:54Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:54Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873"} 2026-07-05T06:23:54Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873"} 2026-07-05T06:23:54Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873"} 2026-07-05T06:23:54Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:54Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873"} 2026-07-05T06:23:54Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873"} 2026-07-05T06:23:54Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:54Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:54Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:54Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a29cc19b-4ba2-453c-aadf-a50dc204e873"} 2026-07-05T06:23:59Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:23:59Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:23:59Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:23:59Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db"} 2026-07-05T06:23:59Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db"} 2026-07-05T06:23:59Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db"} 2026-07-05T06:23:59Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:59Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db"} 2026-07-05T06:23:59Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db"} 2026-07-05T06:23:59Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db", "namespace": "redhat-ods-applications"} 2026-07-05T06:23:59Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:23:59Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:23:59Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb004d28-bc69-4b6f-b7c3-24b12c56b4db"} 2026-07-05T06:24:04Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:24:04Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:24:04Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:24:04Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30"} 2026-07-05T06:24:04Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30"} 2026-07-05T06:24:04Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30"} 2026-07-05T06:24:04Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:04Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30"} 2026-07-05T06:24:04Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30"} 2026-07-05T06:24:04Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:04Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:04Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:04Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "fb24acdc-8478-45b3-a1bb-86ae75f82c30"} 2026-07-05T06:24:04Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:24:04Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:24:04Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:24:04Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:24:04Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:24:04Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35"} 2026-07-05T06:24:04Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35"} 2026-07-05T06:24:04Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35"} 2026-07-05T06:24:04Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:04Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35"} 2026-07-05T06:24:04Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35"} 2026-07-05T06:24:04Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:04Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:05Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:05Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2779e77c-6b82-4ed9-a595-5a3f63d7ff35"} 2026-07-05T06:24:09Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:24:09Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:24:09Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:24:09Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f"} 2026-07-05T06:24:09Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f"} 2026-07-05T06:24:09Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f"} 2026-07-05T06:24:09Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:10Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f"} 2026-07-05T06:24:10Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f"} 2026-07-05T06:24:10Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:10Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:10Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:10Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "331b354d-ee2d-4393-9f09-0fed84a9dd9f"} 2026-07-05T06:24:11Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:24:11Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:24:12Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:24:12Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5"} 2026-07-05T06:24:12Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5"} 2026-07-05T06:24:12Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5"} 2026-07-05T06:24:12Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:12Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5"} 2026-07-05T06:24:12Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5"} 2026-07-05T06:24:12Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:12Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:12Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:12Z INFO FeatureStore installation complete {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5"} 2026-07-05T06:24:12Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:24:12Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:24:12Z INFO Added Feast discovery label to namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:24:12Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:12Z INFO Successfully added feature store to namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "namespace": "test-fs-ui-proj-108259", "clientConfig": "feast-test-s3-client", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:12Z DEBUG Registry permissions response {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "body": "{\"permissions\":[],\"pagination\":{}}"} 2026-07-05T06:24:12Z DEBUG Extracted permission policies {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "permissionsCount": 0, "policiesCount": 0} 2026-07-05T06:24:12Z DEBUG Auto-access prerequisites missing or registry unreachable; cleaning up stale auto-access RBAC {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a5ac74b0-2fd0-49e1-954c-6a98b08e69b5", "policies": 0, "clientConfigMapSet": true, "fetchError": false} 2026-07-05T06:24:12Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:24:12Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:24:12Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:24:12Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83"} 2026-07-05T06:24:12Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83"} 2026-07-05T06:24:12Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83"} 2026-07-05T06:24:12Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:12Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83"} 2026-07-05T06:24:12Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83"} 2026-07-05T06:24:12Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:12Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:12Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:13Z INFO FeatureStore installation complete {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83"} 2026-07-05T06:24:13Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:13Z INFO Successfully added feature store to namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "namespace": "test-fs-ui-proj-108259", "clientConfig": "feast-test-s3-client", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:13Z DEBUG Registry permissions response {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "body": "{\"permissions\":[],\"pagination\":{}}"} 2026-07-05T06:24:13Z DEBUG Extracted permission policies {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "permissionsCount": 0, "policiesCount": 0} 2026-07-05T06:24:13Z DEBUG Auto-access prerequisites missing or registry unreachable; cleaning up stale auto-access RBAC {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "e6335b66-7137-4d9b-aa81-c32f1a2b6a83", "policies": 0, "clientConfigMapSet": true, "fetchError": false} 2026-07-05T06:24:15Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:24:15Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:24:15Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:24:15Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a"} 2026-07-05T06:24:15Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a"} 2026-07-05T06:24:15Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a"} 2026-07-05T06:24:15Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:15Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a"} 2026-07-05T06:24:15Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a"} 2026-07-05T06:24:15Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:15Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:15Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:15Z INFO FeatureStore installation complete {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a"} 2026-07-05T06:24:15Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:15Z INFO Successfully added feature store to namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "namespace": "test-fs-ui-proj-108259", "clientConfig": "feast-test-s3-client", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:15Z DEBUG Registry permissions response {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "body": "{\"permissions\":[],\"pagination\":{}}"} 2026-07-05T06:24:15Z DEBUG Extracted permission policies {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "permissionsCount": 0, "policiesCount": 0} 2026-07-05T06:24:15Z DEBUG Auto-access prerequisites missing or registry unreachable; cleaning up stale auto-access RBAC {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "923f4381-ece6-4c69-9d50-a2244bb6971a", "policies": 0, "clientConfigMapSet": true, "fetchError": false} 2026-07-05T06:24:45Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:24:45Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:24:45Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:24:45Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee"} 2026-07-05T06:24:45Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee"} 2026-07-05T06:24:45Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee"} 2026-07-05T06:24:45Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:45Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee"} 2026-07-05T06:24:45Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee"} 2026-07-05T06:24:45Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:45Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:45Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:45Z INFO FeatureStore installation complete {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee"} 2026-07-05T06:24:45Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:45Z INFO Successfully added feature store to namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "namespace": "test-fs-ui-proj-108259", "clientConfig": "feast-test-s3-client", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:45Z DEBUG Registry permissions response {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "body": "{\"permissions\":[],\"pagination\":{}}"} 2026-07-05T06:24:45Z DEBUG Extracted permission policies {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "permissionsCount": 0, "policiesCount": 0} 2026-07-05T06:24:45Z DEBUG Auto-access prerequisites missing or registry unreachable; cleaning up stale auto-access RBAC {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "6fad5758-2dd4-40c7-ad5f-2de58c79c6ee", "policies": 0, "clientConfigMapSet": true, "fetchError": false} 2026-07-05T06:24:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:24:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:24:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:24:46Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52"} 2026-07-05T06:24:46Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52"} 2026-07-05T06:24:46Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52"} 2026-07-05T06:24:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:46Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52"} 2026-07-05T06:24:46Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52"} 2026-07-05T06:24:46Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:46Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:46Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "86ab367e-272a-4b58-ad39-3c7e352d8a52"} 2026-07-05T06:24:46Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:24:46Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:24:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:24:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:24:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:24:46Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580"} 2026-07-05T06:24:47Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580"} 2026-07-05T06:24:47Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580"} 2026-07-05T06:24:47Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:47Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580"} 2026-07-05T06:24:47Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580"} 2026-07-05T06:24:47Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:47Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:47Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:47Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bdc3e567-3193-47f6-a092-5be8065a5580"} 2026-07-05T06:24:51Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:24:51Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:24:51Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:24:51Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc"} 2026-07-05T06:24:51Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc"} 2026-07-05T06:24:51Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc"} 2026-07-05T06:24:51Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:51Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc"} 2026-07-05T06:24:51Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc"} 2026-07-05T06:24:51Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:51Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:52Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:52Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3d058260-f091-48b4-a6a1-ed5681d092cc"} 2026-07-05T06:24:57Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:24:57Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:24:57Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:24:57Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3"} 2026-07-05T06:24:57Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3"} 2026-07-05T06:24:57Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3"} 2026-07-05T06:24:57Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:57Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3"} 2026-07-05T06:24:57Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3"} 2026-07-05T06:24:57Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3", "namespace": "redhat-ods-applications"} 2026-07-05T06:24:57Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:24:57Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:24:57Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "101c4b9b-7b0d-4c53-b080-9569d28f36a3"} 2026-07-05T06:25:02Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:02Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:02Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:02Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525"} 2026-07-05T06:25:02Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525"} 2026-07-05T06:25:02Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525"} 2026-07-05T06:25:02Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:02Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525"} 2026-07-05T06:25:02Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525"} 2026-07-05T06:25:02Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:02Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:02Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:02Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2009c60a-3848-46e1-829f-3a5275815525"} 2026-07-05T06:25:07Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:07Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:07Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:07Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2"} 2026-07-05T06:25:07Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2"} 2026-07-05T06:25:07Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2"} 2026-07-05T06:25:07Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:07Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2"} 2026-07-05T06:25:07Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2"} 2026-07-05T06:25:07Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:07Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:08Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:08Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "97eaff9c-dfe7-44af-a139-4e0237ac76b2"} 2026-07-05T06:25:13Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:13Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:13Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:13Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845"} 2026-07-05T06:25:13Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845"} 2026-07-05T06:25:13Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845"} 2026-07-05T06:25:13Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:13Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845"} 2026-07-05T06:25:13Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845"} 2026-07-05T06:25:13Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:13Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:13Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:13Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "01590879-b0f8-4932-a276-acbac1175845"} 2026-07-05T06:25:15Z DEBUG Notebook is being deleted, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "620bd5c4-f0c1-4eb5-b583-88518082983b"} 2026-07-05T06:25:15Z DEBUG Notebook not found, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "daf2a4a5-0a5e-4053-84fd-8e4355e3eb2f"} 2026-07-05T06:25:18Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:18Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:18Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:18Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311"} 2026-07-05T06:25:18Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311"} 2026-07-05T06:25:18Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311"} 2026-07-05T06:25:18Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:18Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311"} 2026-07-05T06:25:18Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311"} 2026-07-05T06:25:18Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:18Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:18Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:18Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ac51848-c1b4-4d19-96df-666165ee9311"} 2026-07-05T06:25:21Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "9b4e380f-9981-45cb-ac98-8f3c7ff2e181", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:21Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "67e785f1-a47e-4917-ad15-190d4b2c1aae", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:21Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "30cb0aab-2ea5-46a2-9439-1b7ad6181b21", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:22Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "79d414e0-bc2f-496c-b3a0-cac8aa3cafdc", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:22Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "235b7ff2-0ec7-4f63-aca1-67bf601e37be", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:22Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "c0e97b5f-435c-454d-94bd-914e86be48eb", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d965dbc7-6df9-45af-866c-7cfeb7bce246", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d965dbc7-6df9-45af-866c-7cfeb7bce246", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d965dbc7-6df9-45af-866c-7cfeb7bce246", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:23Z INFO Error: configmaps "feast-test-s3-intra-comm" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d965dbc7-6df9-45af-866c-7cfeb7bce246"} 2026-07-05T06:25:23Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d965dbc7-6df9-45af-866c-7cfeb7bce246"} 2026-07-05T06:25:23Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d965dbc7-6df9-45af-866c-7cfeb7bce246", "error": "configmaps \"feast-test-s3-intra-comm\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:23Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:23Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:23Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bb97a44a-531a-418b-95fd-42e5b7ed34bc"} 2026-07-05T06:25:23Z INFO FeatureStore object modified, retry syncing status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bb97a44a-531a-418b-95fd-42e5b7ed34bc"} 2026-07-05T06:25:23Z ERROR Error updating the FeatureStore status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bb97a44a-531a-418b-95fd-42e5b7ed34bc", "error": "Operation cannot be fulfilled on featurestores.feast.dev \"test-s3\": the object has been modified; please apply your changes to the latest version and try again"} github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:114 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:23Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bb97a44a-531a-418b-95fd-42e5b7ed34bc"} 2026-07-05T06:25:23Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bb97a44a-531a-418b-95fd-42e5b7ed34bc", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:24Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "390623fa-8dc1-4fb3-aeea-27e3d3fb960c"} 2026-07-05T06:25:24Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "390623fa-8dc1-4fb3-aeea-27e3d3fb960c"} 2026-07-05T06:25:24Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "390623fa-8dc1-4fb3-aeea-27e3d3fb960c", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:24Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:24Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:24Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af8d827c-8d80-449f-90f6-1cf8136979af"} 2026-07-05T06:25:24Z INFO FeatureStore object modified, retry syncing status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af8d827c-8d80-449f-90f6-1cf8136979af"} 2026-07-05T06:25:24Z ERROR Error updating the FeatureStore status {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af8d827c-8d80-449f-90f6-1cf8136979af", "error": "Operation cannot be fulfilled on featurestores.feast.dev \"test-s3\": the object has been modified; please apply your changes to the latest version and try again"} github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:114 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:24Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af8d827c-8d80-449f-90f6-1cf8136979af"} 2026-07-05T06:25:24Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "af8d827c-8d80-449f-90f6-1cf8136979af", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:24Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5bd548fb-dab8-490f-9237-33af0c438aa5"} 2026-07-05T06:25:24Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5bd548fb-dab8-490f-9237-33af0c438aa5"} 2026-07-05T06:25:24Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5bd548fb-dab8-490f-9237-33af0c438aa5", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:24Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "dfed4b66-d71b-420b-9a7f-efefe04ac6c4"} 2026-07-05T06:25:24Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "dfed4b66-d71b-420b-9a7f-efefe04ac6c4"} 2026-07-05T06:25:24Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "dfed4b66-d71b-420b-9a7f-efefe04ac6c4", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:24Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c2e58771-3bc4-4f29-af36-d0d3067370da"} 2026-07-05T06:25:24Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c2e58771-3bc4-4f29-af36-d0d3067370da"} 2026-07-05T06:25:24Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c2e58771-3bc4-4f29-af36-d0d3067370da", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:24Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "98430d00-267a-478d-beb8-03937abfd034"} 2026-07-05T06:25:24Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "98430d00-267a-478d-beb8-03937abfd034"} 2026-07-05T06:25:24Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "98430d00-267a-478d-beb8-03937abfd034", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:24Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a9b99ca8-a583-45b4-a403-0d146ce909cc"} 2026-07-05T06:25:24Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a9b99ca8-a583-45b4-a403-0d146ce909cc"} 2026-07-05T06:25:24Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a9b99ca8-a583-45b4-a403-0d146ce909cc", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:24Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9f971c20-bfd6-4b03-b044-7dafc1670d3e"} 2026-07-05T06:25:24Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9f971c20-bfd6-4b03-b044-7dafc1670d3e"} 2026-07-05T06:25:24Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9f971c20-bfd6-4b03-b044-7dafc1670d3e", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:24Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ebadf0d-16f6-4f89-8bfc-007933d28af1"} 2026-07-05T06:25:24Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ebadf0d-16f6-4f89-8bfc-007933d28af1"} 2026-07-05T06:25:24Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9ebadf0d-16f6-4f89-8bfc-007933d28af1", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:25Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "521b1807-2d8a-4a6c-ab98-eea38734c348"} 2026-07-05T06:25:25Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "521b1807-2d8a-4a6c-ab98-eea38734c348"} 2026-07-05T06:25:25Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "521b1807-2d8a-4a6c-ab98-eea38734c348", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:25Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "da6cf821-7404-4a2c-90e9-ea96bb839e54"} 2026-07-05T06:25:25Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "da6cf821-7404-4a2c-90e9-ea96bb839e54"} 2026-07-05T06:25:25Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "da6cf821-7404-4a2c-90e9-ea96bb839e54", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:25Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8a3d91d7-cbf1-4f2e-92b8-1d4879c09fcf"} 2026-07-05T06:25:25Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8a3d91d7-cbf1-4f2e-92b8-1d4879c09fcf"} 2026-07-05T06:25:25Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8a3d91d7-cbf1-4f2e-92b8-1d4879c09fcf", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:25Z INFO Error: configmaps "feast-test-s3-client-ca" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bade4021-dd4f-4d3f-b6b8-d21f08d5aa98"} 2026-07-05T06:25:25Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bade4021-dd4f-4d3f-b6b8-d21f08d5aa98"} 2026-07-05T06:25:25Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "bade4021-dd4f-4d3f-b6b8-d21f08d5aa98", "error": "configmaps \"feast-test-s3-client-ca\" is forbidden: unable to create new content in namespace test-fs-ui-proj-108259 because it is being terminated"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:25:26Z DEBUG FeatureStore CR not found, has been deleted {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1930b9c0-d596-4b0e-b32f-2a1823f4969a"} 2026-07-05T06:25:26Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:26Z INFO Removed Feast discovery label from namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1930b9c0-d596-4b0e-b32f-2a1823f4969a", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:25:26Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1930b9c0-d596-4b0e-b32f-2a1823f4969a", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:26Z INFO Removing feature store from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1930b9c0-d596-4b0e-b32f-2a1823f4969a", "featureStoreName": "test-s3", "featureStoreNamespace": "test-fs-ui-proj-108259", "clientConfigName": "", "expectedClientConfigName": "feast-test-s3-client"} 2026-07-05T06:25:26Z INFO Removing config from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1930b9c0-d596-4b0e-b32f-2a1823f4969a", "config": "feast-test-s3-client"} 2026-07-05T06:25:26Z INFO Removed empty namespace entry from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1930b9c0-d596-4b0e-b32f-2a1823f4969a", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:25:26Z INFO Updated namespace registry ConfigMap {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1930b9c0-d596-4b0e-b32f-2a1823f4969a", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "remainingConfigs": [], "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:26Z INFO Successfully removed feature store from namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1930b9c0-d596-4b0e-b32f-2a1823f4969a", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:26Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "d0067b93-3580-477a-9853-f80731513671", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:27Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "0ab02121-e5c6-4efb-aced-4a914ceec1b0", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:29Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "ConfigMap": "feast-test-s3-client-ca", "operation": "created"} 2026-07-05T06:25:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "Service": "feast-test-s3-online", "operation": "created"} 2026-07-05T06:25:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "Service": "feast-test-s3-registry", "operation": "created"} 2026-07-05T06:25:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "Service": "feast-test-s3-registry-rest", "operation": "created"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "ServiceAccount": "feast-test-s3", "operation": "created"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "ConfigMap": "feast-test-s3-intra-comm", "operation": "created"} 2026-07-05T06:25:30Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e"} 2026-07-05T06:25:30Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e"} 2026-07-05T06:25:30Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "Deployment": "feast-test-s3", "operation": "created"} 2026-07-05T06:25:30Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e"} 2026-07-05T06:25:30Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "ConfigMap": "feast-test-s3-client", "operation": "created"} 2026-07-05T06:25:30Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:30Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:30Z INFO Successfully reconciled namespace registry ConfigMap {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "ConfigMap": "feast-configs-registry", "Namespace": "redhat-ods-applications", "operation": "updated"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "Role": "feast-test-s3-cronjob", "operation": "created"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "RoleBinding": "feast-test-s3-cronjob", "operation": "created"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e", "CronJob": "feast-test-s3", "operation": "created"} 2026-07-05T06:25:30Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "d4aae87b-c964-4306-b41f-df6da578136e"} 2026-07-05T06:25:30Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:30Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:30Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268"} 2026-07-05T06:25:30Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268"} 2026-07-05T06:25:30Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:30Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268"} 2026-07-05T06:25:30Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268"} 2026-07-05T06:25:30Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:30Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:30Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:30Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "2a1db0d4-7e5c-4ece-978a-880a94e27268"} 2026-07-05T06:25:34Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "6cc23d27-525d-4f6b-8de6-41552a68eb76", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:35Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:35Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:35Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:35Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1"} 2026-07-05T06:25:35Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1"} 2026-07-05T06:25:35Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1"} 2026-07-05T06:25:35Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:35Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1"} 2026-07-05T06:25:35Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1"} 2026-07-05T06:25:35Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:35Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:35Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:35Z INFO Deployment is not available: Container 'online' waiting: PodInitializing {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c70a8772-a234-4c73-893a-4d47e65621b1"} 2026-07-05T06:25:39Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "ef405715-7612-422c-8965-b5572abe3439", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:39Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "1e9378f8-eb8f-44db-bae1-e51f2dcbd61f", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:40Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:40Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a"} 2026-07-05T06:25:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a"} 2026-07-05T06:25:41Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a"} 2026-07-05T06:25:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a"} 2026-07-05T06:25:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a"} 2026-07-05T06:25:41Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:41Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:41Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8974171a-1299-4a3a-a581-3a2c27b4a61a"} 2026-07-05T06:25:41Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:41Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5"} 2026-07-05T06:25:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5"} 2026-07-05T06:25:41Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5"} 2026-07-05T06:25:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5"} 2026-07-05T06:25:41Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5"} 2026-07-05T06:25:41Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:41Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:41Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:41Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b12545bf-a683-4663-8fb3-037769a872f5"} 2026-07-05T06:25:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:46Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70"} 2026-07-05T06:25:46Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70"} 2026-07-05T06:25:46Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70"} 2026-07-05T06:25:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:46Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70"} 2026-07-05T06:25:46Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70"} 2026-07-05T06:25:46Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:46Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:46Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:46Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "5094f64c-0524-4230-aa70-cb736e130c70"} 2026-07-05T06:25:48Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:48Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:48Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:48Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0"} 2026-07-05T06:25:48Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0"} 2026-07-05T06:25:48Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0"} 2026-07-05T06:25:48Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:48Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0"} 2026-07-05T06:25:48Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0"} 2026-07-05T06:25:48Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:48Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:49Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:49Z INFO FeatureStore installation complete {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0"} 2026-07-05T06:25:49Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:49Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:25:49Z INFO Added Feast discovery label to namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:25:49Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:49Z INFO Successfully added feature store to namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "namespace": "test-fs-ui-proj-108259", "clientConfig": "feast-test-s3-client", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:49Z DEBUG Registry permissions response {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "body": "{\"permissions\":[],\"pagination\":{}}"} 2026-07-05T06:25:49Z DEBUG Extracted permission policies {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "permissionsCount": 0, "policiesCount": 0} 2026-07-05T06:25:49Z DEBUG Auto-access prerequisites missing or registry unreachable; cleaning up stale auto-access RBAC {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "25773d5a-fe78-40fd-b47d-b1f4457ccda0", "policies": 0, "clientConfigMapSet": true, "fetchError": false} 2026-07-05T06:25:49Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:49Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:49Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:49Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321"} 2026-07-05T06:25:49Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321"} 2026-07-05T06:25:49Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321"} 2026-07-05T06:25:49Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:49Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321"} 2026-07-05T06:25:49Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321"} 2026-07-05T06:25:49Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:49Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:49Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:49Z INFO FeatureStore installation complete {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321"} 2026-07-05T06:25:49Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:49Z INFO Successfully added feature store to namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "namespace": "test-fs-ui-proj-108259", "clientConfig": "feast-test-s3-client", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:49Z DEBUG Registry permissions response {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "body": "{\"permissions\":[],\"pagination\":{}}"} 2026-07-05T06:25:49Z DEBUG Extracted permission policies {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "permissionsCount": 0, "policiesCount": 0} 2026-07-05T06:25:49Z DEBUG Auto-access prerequisites missing or registry unreachable; cleaning up stale auto-access RBAC {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3112ce09-74f0-427a-8e05-3f47b3239321", "policies": 0, "clientConfigMapSet": true, "fetchError": false} 2026-07-05T06:25:51Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:25:51Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:25:51Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:25:51Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38"} 2026-07-05T06:25:51Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38"} 2026-07-05T06:25:51Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38"} 2026-07-05T06:25:51Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:51Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38"} 2026-07-05T06:25:51Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38"} 2026-07-05T06:25:51Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:51Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:51Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:25:51Z INFO FeatureStore installation complete {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38"} 2026-07-05T06:25:51Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "namespace": "redhat-ods-applications"} 2026-07-05T06:25:51Z INFO Successfully added feature store to namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "namespace": "test-fs-ui-proj-108259", "clientConfig": "feast-test-s3-client", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:25:51Z DEBUG Registry permissions response {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "body": "{\"permissions\":[],\"pagination\":{}}"} 2026-07-05T06:25:51Z DEBUG Extracted permission policies {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "permissionsCount": 0, "policiesCount": 0} 2026-07-05T06:25:51Z DEBUG Auto-access prerequisites missing or registry unreachable; cleaning up stale auto-access RBAC {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "22008006-c706-40d6-8930-3c6addd13f38", "policies": 0, "clientConfigMapSet": true, "fetchError": false} 2026-07-05T06:25:58Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "f11ffe2b-a559-4d8f-8f7d-2ef422226f77", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:58Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "2e4da768-9c9d-45a0-bfa6-6f3e3788a3f4", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:58Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "51be9527-7cdd-4f78-b906-09c8b401b680", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:25:59Z DEBUG Notebook is being deleted, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "5247c407-0687-4f84-9941-911bd4a5af08"} 2026-07-05T06:25:59Z DEBUG Notebook not found, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"jupyter-nb-htpasswd-2dcluster-2dadmin-2duser","namespace":"rhods-notebooks"}, "namespace": "rhods-notebooks", "name": "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser", "reconcileID": "0a8331b8-651d-43d6-b512-d89b88040e24"} 2026-07-05T06:26:07Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:26:07Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:26:07Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:26:08Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e"} 2026-07-05T06:26:08Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e"} 2026-07-05T06:26:08Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e"} 2026-07-05T06:26:08Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:08Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e"} 2026-07-05T06:26:08Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e"} 2026-07-05T06:26:08Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:08Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:08Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:08Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8cfebc61-239a-47c3-8331-adb2f1e8792e"} 2026-07-05T06:26:08Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:26:08Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:26:08Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:26:08Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:26:08Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:26:08Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1"} 2026-07-05T06:26:08Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1"} 2026-07-05T06:26:08Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1"} 2026-07-05T06:26:08Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:08Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1"} 2026-07-05T06:26:08Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1"} 2026-07-05T06:26:08Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:08Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:08Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:08Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c98420fa-8627-49d9-b74a-a0d2698956d1"} 2026-07-05T06:26:13Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:26:13Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:26:13Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:26:13Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2"} 2026-07-05T06:26:13Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2"} 2026-07-05T06:26:13Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2"} 2026-07-05T06:26:13Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:13Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2"} 2026-07-05T06:26:13Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2"} 2026-07-05T06:26:13Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:13Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:13Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:13Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c799a213-0a40-4e83-9d2e-1ac99e546ea2"} 2026-07-05T06:26:18Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:26:18Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:26:18Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:26:18Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb"} 2026-07-05T06:26:18Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb"} 2026-07-05T06:26:18Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb"} 2026-07-05T06:26:18Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:18Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb"} 2026-07-05T06:26:18Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb"} 2026-07-05T06:26:18Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:18Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:18Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:18Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "a7c40a1b-784a-4960-bd8d-66c32cc1d3cb"} 2026-07-05T06:26:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:26:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:26:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:26:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c"} 2026-07-05T06:26:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c"} 2026-07-05T06:26:23Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c"} 2026-07-05T06:26:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c"} 2026-07-05T06:26:23Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c"} 2026-07-05T06:26:23Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:23Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:23Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:23Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f3c6d886-b2ac-42f2-8419-d67ce4d00a4c"} 2026-07-05T06:26:28Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:26:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:26:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:26:29Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a"} 2026-07-05T06:26:29Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a"} 2026-07-05T06:26:29Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a"} 2026-07-05T06:26:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:29Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a"} 2026-07-05T06:26:29Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a"} 2026-07-05T06:26:29Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:29Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:29Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:29Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7af373cf-71e6-4de9-9172-5ced09949c3a"} 2026-07-05T06:26:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:26:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:26:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:26:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3"} 2026-07-05T06:26:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3"} 2026-07-05T06:26:34Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3"} 2026-07-05T06:26:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3"} 2026-07-05T06:26:34Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3"} 2026-07-05T06:26:34Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:34Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:34Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:34Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "c7177040-039d-4d86-ae19-bb20c1028ad3"} 2026-07-05T06:26:39Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:26:39Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:26:39Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:26:39Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c"} 2026-07-05T06:26:39Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c"} 2026-07-05T06:26:39Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c"} 2026-07-05T06:26:39Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:39Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c"} 2026-07-05T06:26:39Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c"} 2026-07-05T06:26:39Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:39Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:39Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:39Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "32f7922e-c1e8-4d73-9280-0ad1933aba1c"} 2026-07-05T06:26:44Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe", "Service": "feast-test-s3-online", "operation": "updated"} 2026-07-05T06:26:44Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe", "Service": "feast-test-s3-registry", "operation": "updated"} 2026-07-05T06:26:44Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe", "Service": "feast-test-s3-registry-rest", "operation": "updated"} 2026-07-05T06:26:44Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe"} 2026-07-05T06:26:44Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe"} 2026-07-05T06:26:44Z INFO Mounted custom CA bundle ConfigMap to Feast pods. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe"} 2026-07-05T06:26:44Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe", "Deployment": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:44Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe"} 2026-07-05T06:26:45Z INFO Found trusted CA bundle ConfigMap. Using custom CA bundle. {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe"} 2026-07-05T06:26:45Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:45Z DEBUG Deploying namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:45Z INFO Successfully reconciled {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe", "CronJob": "feast-test-s3", "operation": "updated"} 2026-07-05T06:26:45Z INFO Deployment is not available {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "82e0a0ee-8241-4c90-95a8-bd91f7b33bbe"} 2026-07-05T06:26:45Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f148bdde-2334-4d40-8af8-fdc42918f47b", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f148bdde-2334-4d40-8af8-fdc42918f47b"} 2026-07-05T06:26:45Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f148bdde-2334-4d40-8af8-fdc42918f47b"} 2026-07-05T06:26:45Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "f148bdde-2334-4d40-8af8-fdc42918f47b", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:26:45Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:26:45Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b70d0d75-add1-434f-a235-b55bcb9729c5", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b70d0d75-add1-434f-a235-b55bcb9729c5"} 2026-07-05T06:26:45Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b70d0d75-add1-434f-a235-b55bcb9729c5"} 2026-07-05T06:26:45Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "b70d0d75-add1-434f-a235-b55bcb9729c5", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8996ef1c-866e-4c2a-9350-c31554d9f85d", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8996ef1c-866e-4c2a-9350-c31554d9f85d"} 2026-07-05T06:26:45Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8996ef1c-866e-4c2a-9350-c31554d9f85d"} 2026-07-05T06:26:45Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "8996ef1c-866e-4c2a-9350-c31554d9f85d", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ff02ec9e-587d-454a-9f75-fc3aee464afc", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ff02ec9e-587d-454a-9f75-fc3aee464afc"} 2026-07-05T06:26:45Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ff02ec9e-587d-454a-9f75-fc3aee464afc"} 2026-07-05T06:26:45Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ff02ec9e-587d-454a-9f75-fc3aee464afc", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "58519ea7-a14c-4a3d-ba7e-a977804410f3", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "58519ea7-a14c-4a3d-ba7e-a977804410f3"} 2026-07-05T06:26:45Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "58519ea7-a14c-4a3d-ba7e-a977804410f3"} 2026-07-05T06:26:45Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "58519ea7-a14c-4a3d-ba7e-a977804410f3", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "eec9cf84-3fe5-4741-8ef7-1df03b02b51e", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "eec9cf84-3fe5-4741-8ef7-1df03b02b51e"} 2026-07-05T06:26:45Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "eec9cf84-3fe5-4741-8ef7-1df03b02b51e"} 2026-07-05T06:26:45Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "eec9cf84-3fe5-4741-8ef7-1df03b02b51e", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ff5db45e-df59-4061-a59e-d21687f2ba17", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ff5db45e-df59-4061-a59e-d21687f2ba17"} 2026-07-05T06:26:45Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ff5db45e-df59-4061-a59e-d21687f2ba17"} 2026-07-05T06:26:45Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ff5db45e-df59-4061-a59e-d21687f2ba17", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3e2e1c25-62ac-46fc-b773-552c46dad460", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3e2e1c25-62ac-46fc-b773-552c46dad460"} 2026-07-05T06:26:45Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3e2e1c25-62ac-46fc-b773-552c46dad460"} 2026-07-05T06:26:45Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "3e2e1c25-62ac-46fc-b773-552c46dad460", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1f8aae06-cf26-4f21-b615-93a3aa190ea7", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:45Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1f8aae06-cf26-4f21-b615-93a3aa190ea7"} 2026-07-05T06:26:45Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1f8aae06-cf26-4f21-b615-93a3aa190ea7"} 2026-07-05T06:26:45Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "1f8aae06-cf26-4f21-b615-93a3aa190ea7", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:46Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ea90da68-4263-42b1-adbd-8739790c8a20", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:46Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ea90da68-4263-42b1-adbd-8739790c8a20"} 2026-07-05T06:26:46Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ea90da68-4263-42b1-adbd-8739790c8a20"} 2026-07-05T06:26:46Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "ea90da68-4263-42b1-adbd-8739790c8a20", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:46Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "369ae23f-9fc4-4e84-845e-e1120ae63c98", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:46Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "369ae23f-9fc4-4e84-845e-e1120ae63c98"} 2026-07-05T06:26:46Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "369ae23f-9fc4-4e84-845e-e1120ae63c98"} 2026-07-05T06:26:46Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "369ae23f-9fc4-4e84-845e-e1120ae63c98", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:46Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "0aad855d-69a2-4e31-8cd3-80a973d11cca", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:46Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "0aad855d-69a2-4e31-8cd3-80a973d11cca"} 2026-07-05T06:26:46Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "0aad855d-69a2-4e31-8cd3-80a973d11cca"} 2026-07-05T06:26:46Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "0aad855d-69a2-4e31-8cd3-80a973d11cca", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:46Z ERROR invalid secret feast-online-store-secret for offline store {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9a25b73a-64b0-4b57-b37b-3245b1760160", "error": "secrets \"feast-online-store-secret\" not found"} github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).getSecret /opt/app-root/src/internal/controller/services/util.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).validateOnlineStorePersistence /opt/app-root/src/internal/controller/services/services.go:206 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).reconcileServices /opt/app-root/src/internal/controller/services/services.go:130 github.com/feast-dev/feast/infra/feast-operator/internal/controller/services.(*FeastServices).Deploy /opt/app-root/src/internal/controller/services/services.go:76 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).deployFeast /opt/app-root/src/internal/controller/featurestore_controller.go:294 github.com/feast-dev/feast/infra/feast-operator/internal/controller.(*FeatureStoreReconciler).Reconcile /opt/app-root/src/internal/controller/featurestore_controller.go:106 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:119 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:340 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:46Z INFO Error: secrets "feast-online-store-secret" not found {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9a25b73a-64b0-4b57-b37b-3245b1760160"} 2026-07-05T06:26:46Z INFO Warning: Reconciler returned both a non-zero result and a non-nil error. The result will always be ignored if the error is non-nil and the non-nil error causes requeuing with exponential backoff. For more details, see: https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/reconcile#Reconciler {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9a25b73a-64b0-4b57-b37b-3245b1760160"} 2026-07-05T06:26:46Z ERROR Reconciler error {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "9a25b73a-64b0-4b57-b37b-3245b1760160", "error": "secrets \"feast-online-store-secret\" not found"} sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:353 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:300 sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func2.1 /cachi2/output/deps/gomod/pkg/mod/sigs.k8s.io/controller-runtime@v0.21.0/pkg/internal/controller/controller.go:202 2026-07-05T06:26:46Z DEBUG FeatureStore changed, finding affected Notebooks {"project": "credit_scoring_local", "featurestore": "test-fs-ui-proj-108259/test-s3"} 2026-07-05T06:26:46Z DEBUG FeatureStore CR not found, has been deleted {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "80f9c233-6fcf-4c9e-b2c7-57329ac8bb55"} 2026-07-05T06:26:46Z INFO Removed Feast discovery label from namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "80f9c233-6fcf-4c9e-b2c7-57329ac8bb55", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:26:47Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "80f9c233-6fcf-4c9e-b2c7-57329ac8bb55", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:47Z INFO Removing feature store from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "80f9c233-6fcf-4c9e-b2c7-57329ac8bb55", "featureStoreName": "test-s3", "featureStoreNamespace": "test-fs-ui-proj-108259", "clientConfigName": "", "expectedClientConfigName": "feast-test-s3-client"} 2026-07-05T06:26:47Z INFO Removing config from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "80f9c233-6fcf-4c9e-b2c7-57329ac8bb55", "config": "feast-test-s3-client"} 2026-07-05T06:26:47Z INFO Removed empty namespace entry from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "80f9c233-6fcf-4c9e-b2c7-57329ac8bb55", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:26:47Z INFO Updated namespace registry ConfigMap {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "80f9c233-6fcf-4c9e-b2c7-57329ac8bb55", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "remainingConfigs": [], "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:47Z INFO Successfully removed feature store from namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "80f9c233-6fcf-4c9e-b2c7-57329ac8bb55", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:47Z DEBUG FeatureStore CR not found, has been deleted {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7eb990aa-67b6-4de7-891f-a794f8d89376"} 2026-07-05T06:26:47Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7eb990aa-67b6-4de7-891f-a794f8d89376", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:47Z INFO Removing feature store from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7eb990aa-67b6-4de7-891f-a794f8d89376", "featureStoreName": "test-s3", "featureStoreNamespace": "test-fs-ui-proj-108259", "clientConfigName": "", "expectedClientConfigName": "feast-test-s3-client"} 2026-07-05T06:26:47Z DEBUG Namespace not found in registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7eb990aa-67b6-4de7-891f-a794f8d89376", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:26:47Z INFO Updated namespace registry ConfigMap {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7eb990aa-67b6-4de7-891f-a794f8d89376", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "remainingConfigs": [], "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:47Z INFO Successfully removed feature store from namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "7eb990aa-67b6-4de7-891f-a794f8d89376", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:48Z DEBUG FeatureStore CR not found, has been deleted {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4c267bce-1a41-44a1-8d4c-edd73599e560"} 2026-07-05T06:26:48Z DEBUG Using OpenShift namespace {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4c267bce-1a41-44a1-8d4c-edd73599e560", "namespace": "redhat-ods-applications"} 2026-07-05T06:26:48Z INFO Removing feature store from registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4c267bce-1a41-44a1-8d4c-edd73599e560", "featureStoreName": "test-s3", "featureStoreNamespace": "test-fs-ui-proj-108259", "clientConfigName": "", "expectedClientConfigName": "feast-test-s3-client"} 2026-07-05T06:26:48Z DEBUG Namespace not found in registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4c267bce-1a41-44a1-8d4c-edd73599e560", "namespace": "test-fs-ui-proj-108259"} 2026-07-05T06:26:48Z INFO Updated namespace registry ConfigMap {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4c267bce-1a41-44a1-8d4c-edd73599e560", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "remainingConfigs": [], "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:26:48Z INFO Successfully removed feature store from namespace registry {"controller": "featurestore", "controllerGroup": "feast.dev", "controllerKind": "FeatureStore", "FeatureStore": {"name":"test-s3","namespace":"test-fs-ui-proj-108259"}, "namespace": "test-fs-ui-proj-108259", "name": "test-s3", "reconcileID": "4c267bce-1a41-44a1-8d4c-edd73599e560", "namespace": "test-fs-ui-proj-108259", "clientConfig": "", "targetNamespace": "redhat-ods-applications"} 2026-07-05T06:30:25Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "d27544bf-5d13-4cf4-8df3-ca18c9e13151", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:30:25Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "6b217b9a-fe94-423a-8e26-5ad76d0d6a03", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:30:25Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "a145cdcc-c87c-41d1-83c4-e136fb7fe82a", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:30:26Z DEBUG Feast integration not enabled, skipping reconciliation {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "828a362d-2ca2-4d32-a3cd-79986941e2e1", "label": "opendatahub.io/feast-integration", "value": ""} 2026-07-05T06:30:46Z DEBUG Notebook is being deleted, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "c6c65570-cb1a-42dc-98d2-2c59505b9c86"} 2026-07-05T06:30:46Z DEBUG Notebook not found, ConfigMap will be cleaned up by owner reference {"controller": "notebook", "controllerGroup": "kubeflow.org", "controllerKind": "Notebook", "Notebook": {"name":"wb-rwo-test","namespace":"test-workbench-storage-preset-167984"}, "namespace": "test-workbench-storage-preset-167984", "name": "wb-rwo-test", "reconcileID": "db79c05c-6297-421a-9c40-3e607d42197d"} ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-76db8fd49f-mclfn.log ==================== 2026/07/05 05:36:51 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/mcp_catalog/v1alpha1/mcp_servers?pageSize=1000 HTTP/1.1" from 127.0.0.1:50140 - 200 98856B in 7.285025ms 2026/07/05 05:36:51 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/mcp_catalog/v1alpha1/mcp_servers?sourceLabel=null HTTP/1.1" from 127.0.0.1:50140 - 200 29701B in 2.761336ms 2026/07/05 05:36:51 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/mcp_catalog/v1alpha1/mcp_servers HTTP/1.1" from 127.0.0.1:50140 - 200 98854B in 5.917876ms 2026/07/05 05:36:51 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/mcp_catalog/v1alpha1/mcp_servers?sourceLabel=Red+Hat HTTP/1.1" from 127.0.0.1:50140 - 200 23678B in 2.580405ms 2026/07/05 05:36:51 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/mcp_catalog/v1alpha1/mcp_servers?sourceLabel=Red+Hat+Partners HTTP/1.1" from 127.0.0.1:50140 - 200 45582B in 4.631701ms 2026/07/05 05:36:52 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/model_catalog/v1alpha1/labels?assetType=models HTTP/1.1" from 127.0.0.1:50140 - 200 472B in 60.728µs 2026/07/05 05:36:52 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/model_catalog/v1alpha1/labels?assetType=mcp_servers HTTP/1.1" from 127.0.0.1:50140 - 200 353B in 51.554µs 2026/07/05 05:36:52 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:50140 - 200 472B in 52.286µs 2026/07/05 05:36:52 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/model_catalog/v1alpha1/models?pageSize=100&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:50140 - 200 225476B in 7.164867ms 2026/07/05 05:36:52 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/model_catalog/v1alpha1/models?pageSize=100&sourceLabel=Red+Hat+AI+Validated HTTP/1.1" from 127.0.0.1:50140 - 200 1983912B in 58.163046ms 2026/07/05 05:36:52 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/model_catalog/v1alpha1/models?pageSize=100&sourceLabel=null HTTP/1.1" from 127.0.0.1:50140 - 200 108345B in 3.647708ms 2026/07/05 05:36:52 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/model_catalog/v1alpha1/models?pageSize=100 HTTP/1.1" from 127.0.0.1:50140 - 200 2317622B in 65.827546ms 2026/07/05 05:36:53 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com/api/model_catalog/v1alpha1/models?pageSize=100&sourceLabel=Red+Hat+AI+Validated%2CRed+Hat+AI HTTP/1.1" from 127.0.0.1:50140 - 200 2209332B in 58.569539ms 2026/07/05 06:27:52 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56826 - 200 420B in 1.069024ms 2026/07/05 06:27:55 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56826 - 200 420B in 791.344µs 2026/07/05 06:28:07 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:56826 - 200 420B in 853.382µs 2026/07/05 06:28:24 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:56826 - 200 472B in 60.53µs 2026/07/05 06:28:24 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:59486 - 200 420B in 954.898µs 2026/07/05 06:28:24 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:56826 - 200 8998B in 1.779789ms 2026/07/05 06:28:25 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:59486 - 200 47545B in 2.701093ms 2026/07/05 06:28:25 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:56826 - 200 141549B in 7.05123ms 2026/07/05 06:28:25 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:59486 - 200 83736B in 4.480406ms 2026/07/05 06:28:25 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:59494 - 200 20726B in 24.765552ms 2026/07/05 06:28:25 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:56826 - 200 41125B in 29.666595ms 2026/07/05 06:28:25 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:59504 - 200 56B in 18.024001ms 2026/07/05 06:28:25 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:59510 - 200 62309B in 57.852416ms 2026/07/05 06:28:38 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:56826 - 200 472B in 60.211µs 2026/07/05 06:28:38 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:59494 - 200 420B in 1.140084ms 2026/07/05 06:28:38 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:58188 - 200 8998B in 3.282227ms 2026/07/05 06:28:38 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:59494 - 200 83736B in 3.348441ms 2026/07/05 06:28:38 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:56826 - 200 47545B in 4.834102ms 2026/07/05 06:28:38 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:58198 - 200 141549B in 17.707755ms 2026/07/05 06:28:38 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:59494 - 200 56B in 4.450147ms 2026/07/05 06:28:38 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58220 - 200 20726B in 104.458864ms 2026/07/05 06:28:38 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:56826 - 200 41125B in 111.811794ms 2026/07/05 06:28:38 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58230 - 200 62309B in 126.540475ms 2026/07/05 06:28:58 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:58220 - 200 420B in 1.207185ms 2026/07/05 06:28:58 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:59494 - 200 472B in 50.172µs 2026/07/05 06:28:58 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:58220 - 200 8998B in 2.684073ms 2026/07/05 06:28:58 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:58220 - 200 141549B in 5.995926ms 2026/07/05 06:28:58 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:59494 - 200 47545B in 3.075871ms 2026/07/05 06:28:58 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:43786 - 200 83736B in 4.017515ms 2026/07/05 06:28:59 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58220 - 200 56B in 4.982288ms 2026/07/05 06:28:59 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:59494 - 200 20726B in 78.873856ms 2026/07/05 06:28:59 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58220 - 200 41125B in 77.368277ms 2026/07/05 06:28:59 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:43804 - 200 62309B in 94.296221ms 2026/07/05 06:29:10 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?filterQuery=artifacts.use_case.string_value%3D%27chatbot%27+AND+artifacts.ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=artifacts.ttft_p90.double_value&pageSize=10&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58220 - 200 293828B in 835.945914ms 2026/07/05 06:29:10 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FMinistral-3-3B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58220 - 200 29027B in 34.585919ms 2026/07/05 06:29:10 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:59494 - 200 41125B in 26.819382ms 2026/07/05 06:29:10 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FMinistral-3-14B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58220 - 200 59712B in 115.506438ms 2026/07/05 06:29:10 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgpt-oss-20b/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52608 - 200 52497B in 139.682078ms 2026/07/05 06:29:11 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52610 - 200 62309B in 222.394673ms 2026/07/05 06:29:11 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fsarvam-30b-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:59494 - 200 24427B in 184.773145ms 2026/07/05 06:29:11 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgpt-oss-20b-essential/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52616 - 200 52679B in 318.116723ms 2026/07/05 06:29:11 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FQwen2.5-7B-Instruct-quantized.w8a8/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52622 - 200 35997B in 236.992541ms 2026/07/05 06:29:11 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FQwen3-8B-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52608 - 200 37135B in 141.303629ms 2026/07/05 06:29:11 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FLlama-3.1-8B-Instruct/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:58220 - 200 55242B in 142.288594ms 2026/07/05 06:29:24 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:45136 - 200 472B in 276.539µs 2026/07/05 06:29:24 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:52610 - 200 420B in 1.30973ms 2026/07/05 06:29:24 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:59494 - 200 8998B in 3.286961ms 2026/07/05 06:29:24 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:45136 - 200 47545B in 2.628952ms 2026/07/05 06:29:24 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:52610 - 200 141549B in 6.658622ms 2026/07/05 06:29:24 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:45140 - 200 83736B in 17.282663ms 2026/07/05 06:29:25 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52610 - 200 56B in 4.048788ms 2026/07/05 06:29:25 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:45136 - 200 41125B in 25.720065ms 2026/07/05 06:29:25 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52610 - 200 20726B in 21.716372ms 2026/07/05 06:29:25 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:45156 - 200 62309B in 42.556197ms 2026/07/05 06:29:36 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?filterQuery=artifacts.use_case.string_value%3D%27chatbot%27+AND+artifacts.ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=artifacts.ttft_p90.double_value&pageSize=10&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52610 - 200 293828B in 846.400493ms 2026/07/05 06:29:36 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FMinistral-3-3B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52610 - 200 29027B in 87.79704ms 2026/07/05 06:29:36 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FMinistral-3-14B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:53800 - 200 59712B in 179.26656ms 2026/07/05 06:29:36 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgpt-oss-20b/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:45136 - 200 52497B in 186.873127ms 2026/07/05 06:29:36 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgpt-oss-20b-essential/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:53802 - 200 52679B in 282.975285ms 2026/07/05 06:29:36 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fsarvam-30b-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52610 - 200 24427B in 255.45196ms 2026/07/05 06:29:36 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:53820 - 200 62309B in 365.006202ms 2026/07/05 06:29:36 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:53818 - 200 41125B in 376.477728ms 2026/07/05 06:29:36 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FQwen2.5-7B-Instruct-quantized.w8a8/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:45136 - 200 35997B in 223.341232ms 2026/07/05 06:29:36 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FQwen3-8B-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:53800 - 200 37135B in 222.996399ms 2026/07/05 06:29:36 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FLlama-3.1-8B-Instruct/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:53802 - 200 55242B in 127.773854ms 2026/07/05 06:29:47 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources HTTP/1.1" from 127.0.0.1:53820 - 200 420B in 1.09569ms 2026/07/05 06:29:47 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/labels HTTP/1.1" from 127.0.0.1:53820 - 200 472B in 50.998µs 2026/07/05 06:29:47 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models/filter_options HTTP/1.1" from 127.0.0.1:53820 - 200 8998B in 2.470856ms 2026/07/05 06:29:47 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI HTTP/1.1" from 127.0.0.1:52610 - 200 47545B in 2.57137ms 2026/07/05 06:29:47 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=Red+Hat+AI+validated HTTP/1.1" from 127.0.0.1:53820 - 200 141549B in 7.094355ms 2026/07/05 06:29:47 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/models?pageSize=4&sourceLabel=null HTTP/1.1" from 127.0.0.1:33882 - 200 83736B in 5.695931ms 2026/07/05 06:29:47 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2Fgemma-2-9b-it/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:33906 - 200 56B in 13.940757ms 2026/07/05 06:29:47 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDeepSeek-R1-0528-quantized.w4a16/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:52610 - 200 20726B in 24.100857ms 2026/07/05 06:29:47 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FApertus-8B-Instruct-2509-FP8-dynamic/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:53820 - 200 41125B in 30.264383ms 2026/07/05 06:29:47 "GET http://model-catalog.apps.rosa.nmanos-konflux1.durx.p3.openshiftapps.com:443/api/model_catalog/v1alpha1/sources/redhat_ai_validated_models/models/RedHatAI%2FDevstral-Small-2-24B-Instruct-2512/artifacts/performance?filterQuery=use_case.string_value%3D%27chatbot%27+AND+ttft_p90.double_value+%3C%3D+2231.15348815918&latencyProperty=ttft_p90&orderBy=ttft_p90.double_value&pageSize=999&recommendations=true&sortOrder=ASC&targetRPS=1 HTTP/1.1" from 127.0.0.1:33904 - 200 62309B in 73.991028ms ==================== WORKLOAD POD LOG: rhoai-model-registries/model-catalog-postgres-f9fd8fcdd-kckh5.log ====================